modbus: fix untrusted loop bound in modbus client

Use size_t type where it makes sense and
check if the buffer is large enough before it is used.

Fixes: #33786
Fixes: #33795

Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>