Workload reports metrics even when it fails to run

[rayx]

I observed the issue when trying to run Hadoop workload. Take its start script as an example:

cb_run_hadoop.sh
    -> execute_load_generator()
        -> eval $CMDLINE 2>&1 >> $OUTPUT_FILE
    -> ~/cb_report_app_metrics.py

While there is check in execute_load_generator() on $CMDLINE return code, it doesn't prevent the script continuing to call ~/cb_report_app_metrics.py to report metrics when return code is non-zero. Below is an example of ~/cb_report_app_metrics.py call in that case:

$ /home/ubuntu/cb_report_app_metrics.py throughput::tps iterations::num latency::msec datagen_time:-0:sec datagen_size::records load_id:192:seqnum load_profile:terasort:name load_level:1:load load_duration:60:sec errors:0:num completion_time:-61:sec quiescent_time:8:sec

The code in ~/cb_report_app_metrics.py does have check against empty metric data, so throughput::tps, iterations::num, latency::msec are discarded. The reported metric is like the following:

metrics> db.latest_runtime_app_VM_centos.find()
[
  {
    _id: '6EEF8830-E753-51AA-9C89-DA179B8CB890',
    app_datagen_time: {
      val: '-0',
      units: 'sec',
      avg: 0.08056872037914692,
      max: 1,
      min: -1
    },
    app_load_id: { val: '211', units: 'seqnum' },
    app_load_profile: { val: 'terasort', units: 'name' },
    app_load_level: {
      val: '2',
      units: 'load',
      avg: 1.5734597156398105,
      max: 2,
      min: 1
    },
    app_load_duration: {
      val: '60',
      units: 'sec',
      avg: 61.4218009478673,
      max: 60,
      min: 60
    },
    app_errors: { val: '0', units: 'num', acc: 1 },
    app_completion_time: {
      val: '-60',
      units: 'sec',
      avg: 55.18483412322275,
      max: 61,
      min: -61
    },
    app_quiescent_time: { val: '9', units: 'sec', avg: 9.763033175355451, max: 78, min: 2 },
    time: 1720766641,
    time_cbtool: 1720766641,
    time_h: '07/12/2024 06:44:01 AM UTC',
    time_cbtool_h: '07/12/2024 06:44:01 AM UTC',
    expid: 'EXP-07-12-2024-02-32-38-AM-UTC',
    uuid: '6EEF8830-E753-51AA-9C89-DA179B8CB890'
  }
]

I doubt if this is by design. In my opinion, the metric data is useless and should be skipped. Saving them in metric store is confusing.

BTW, due to the processing in update_avg_acc_max_min() running the above ~/cb_report_app_metrics.py multiple times generates different metrics, which is confusing too.