results.yml should include in progress, errored and aborted tests

[thrix]

Currently, when tmt execution dies in the middle of the plan execution, the results.yml:

• contains only tests which were executed
• all other tests (running or skipped) are missing

This causes issues to the user as it is not possible to easily for consumers of the results.yml (like Testing Farm) to provide a reasonable output.

Blocker for VirtQE and KernelQE.

🔽 TFT-2847

[happz]

Adding an error result for the test that "killed" the execute should be well within our reach.

It was also discussed (briefly, IIRC, feel free to correct me) that tmt could emit placeholder results, with let's say idle outcome, for collected and soon-to-be-executed tests, converting them to actual results as it goes. To me, this sounds like a good idea, and it would be consistent with the error result mentioned above.

[lukaszachy]

Just a note to the 'idle' state: With 'results:custom' the count of expected vs real results might differ as additional tests might be reported (new test names be the name of the test with results: custom appended by reported name).
This is no blocker by any means, just something to document so consumers of results.yaml are not surprised that the number of results after execute step is higher than before.

[psss]

See also related issue #3401 which suggests the report step to be always executed upon failures.