lib: hex: Code guideline fixes

Fixes violations code guideline related with essential types rules:

- Avoid mixing characters with integers
- Ensure computations are done in the destination precision

Signed-off-by: Flavio Ceolin <flavio.ceolin@intel.com>
Signed-off-by: Abramo Bagnara <abramo.bagnara@bugseng.com>