tests: json: improve code coverage

Adds coverage for the uncovered json_calc_encoded_len()
and covers a bunch more error cases.

This gets us up to 90.1% line coverage and 100% function
coverage.

Fixes: #16944
Partial fix for: #16011

Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>