tests: rbtree: Fix test so its actually runs

1. Add a tag in the testcase.yaml to deal with the following error from
sanitycheck:

E: tests/lib/rbtree/testcase.yaml: can't load (skipping):
   <NotMappingError: error code 6: Value: None is not of a
    mapping type: Path: '/'>

2. Reduced the MAX_NODES so the test will build on small memory systems

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>