Generate results.yaml file in progress

[therazix]

This PR changes how the results.yaml file is saved. Previously, the file was saved only once after the execute step was completed, and it contained only the executed tests, omitting any discovered but unexecuted tests.

With this change, a new result outcome pending is introduced, for tests that have been discovered but not yet executed. Additionally, the saving behavior has been updated:

• After the discover step, the results.yaml file is saved for the first time, with pending results for tests that have been discovered but not yet executed.
• The file is then updated after each test execution to reflect the latest results.

Resolves: #3304
Resolves: #3238

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[martinhoyer]

You can count on me to provide a valuable feedback like color choices or saving 0.0000000001% by iterating differently. 🥇

Nitpicks with no value aside, LGTM! Thanks! :)

[lukaszachy]

Two nitpicks, looks great otherwise.

[therazix]

The test /tests/discover/pruning was failing due to an issue with pruning remote repositories when a path is defined. It seems that this function does not update the paths as expected, causing an error when accessing the fmf_id during the creation of pending results.

I don’t believe this issue is related to this PR, so I removed the fmf_id from the pending results in bd2bbd1. We can restore it once the pruning issue is fixed, but I don't want to block this PR on that.

[lukaszachy]

It seems that this function does not update the paths as expected, causing an error when accessing the fmf_id during the creation of pending results.

Ouch, that reminded me we have flow where discover happens in 'prepare' (dist-git-source: true) which means in this case there are no 'pending' results created. I wouldn't block the PR on this though. (And it affects plan splitter as well:/

[happz]

The test /tests/discover/pruning was failing due to an issue with pruning remote repositories when a path is defined. It seems that this function does not update the paths as expected, causing an error when accessing the fmf_id during the creation of pending results.

I don’t believe this issue is related to this PR, so I removed the fmf_id from the pending results in bd2bbd1. We can restore it once the pruning issue is fixed, but I don't want to block this PR on that.

Please, file an issue, and make it stand out by some nice label, I would dare to give it a priority | must to start with - we can always downgrade, but this should not fall under the table.

There should be no technical reason for not having fmf IDs in pending results, tests were discovered already, and they should have fmf IDs. fmf ID is not in results for fun, somebody asked for it to be in there, and while I do not remember who requested it, I bet they will ask why it's missing in pending results as soon as they run into a run with pending results in RP or somewhere ;)

[psss]

I remember during the discussion someone objecting against having pending as the final state to be included in the finished run. I believe the motivation was to prevent confusion when users are reviewing the result and could consider pending as something which is still running. Are we ok with having pending there? I'd say that if the whole plan is clearly marked as finished, pending is just fine there. Thoughts?

[happz]

I remember during the discussion someone objecting against having pending as the final state to be included in the finished run. I believe the motivation was to prevent confusion when users are reviewing the result and could consider pending as something which is still running. Are we ok with having pending there? I'd say that if the whole plan is clearly marked as finished, pending is just fine there. Thoughts?

Presence of pending in the final results.yaml is part of what this issue is about: it's not just to signal which tests are expected to run in the future in RP or whatever other fancy web UI is out there, but also to represent tests that should have been executed but were not, because of test abort or meteor striking the data center. I understand it as a final state - something that was expected to run, but did not - yet, ever, who knows, that depends on whether I'm observing a run in progress or inspecting results of a run where kernel installation failed and aborted the run.

We can pick a different name, to represent this aspect "should run", as pending may sound to some like "we'll get to it" - well, maybe you won't, because of that stupid meteor hit... Or, we can keep pending, find another name, and force tmt to change all pending to much-more-final-outcome before exiting. I never felt the urge to have such a dedicated name, I for one was fine with pending vs a run no longer runs.

[psss]

Thanks for implementing this! Looks good, just a few minor nitpicks.

[therazix]

Please, file an issue, and make it stand out by some nice label, I would dare to give it a priority | must to start with - we can always downgrade, but this should not fall under the table.

Issue created: #3589

[psss]

Rebased, should be ready for merge.