Bluetooth: Fixes length limit in `bt_hex_real`

Since `str` is the output buffer, `bt_hex_real` should limit its output
length according to the `str` size, not the `hex` size.

Signed-off-by: Jiahao Li <reg@ljh.me>