Skip to content
Commit d9f56705 authored by Dmitrii Golovanov's avatar Dmitrii Golovanov Committed by Johan Hedberg
Browse files

ci: elasticsearch: Upload script improvements



Multiple improvements of the `upload_test_results_es.py` script:

 * JSON objects flattening.

   This feature allows `twister.json` file preprocessing to simplify
   its Elasticsearch index structure for complex hierarhical objects,
   for example with memory footprint, or code coverage data.

   A new command line option `--flatten` is added to change testsuite data
   structure in regard of one of its list objects: either `testcases` or
   `recording`, so each item there becomes an independent data record
   inheriting all other testsuite properties, whereas the children
   object's properties are renamed with the parent object's name
   as a prefix: 'testcases_' or 'recording_' respectively.
   Only one testsuite property can be flattened this way per index upload.
   Other children objects will be treated accorging to the index structure.

   Related new command line options (with help text explanations):
    `--flatten-dict-name`,
    `--flatten-list-names`,
    `--flatten-separator`,
    `--transpose-separator`,
    `--escape-separator`

 * A new command line option `--transform` is added to allow regexp group
   parsing in string propertites extracting additional derived properties.

 * A new command line option `--exclude` is added to exclude testsuite
   properties not needed to store at Elasticsearch index.

 * Branch name `--run-branch` and Workflow ID `--run-workflow` command
   line options as additional key fields to allow data from different
   branches, workflows and triggering events in the same index.

 * A new command line option `--map-file` is added to apply
   an explicit index structure to the `twister.json` input data.

 * Add bulk operation timeout parameter for heavy/long uploads.

Other changes:
  * batch upload error handling and logging;
  * inline documentation improvements;
  * some corner case fixes on empty objects.

Signed-off-by: default avatarDmitrii Golovanov <dmitrii.golovanov@intel.com>
parent fa0bbad2
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment