Gitlab pipelines might mark as "broken spec" build that failed because of system failures

[alalazo]

Inspecting the list of pipelines run on develop there is a system failure when building mpich occurring here:

• https://gitlab.spack.io/spack/spack/-/pipelines/173449/failures

The pipeline running after the one above fails, reporting the two mpich as broken specs:

• https://gitlab.spack.io/spack/spack/-/jobs/3334734

spack/spack#32839 swaps two lines of code to trigger a rebuild of mpich, which results in the following jobs for the aws-isc pipeline:

• https://gitlab.spack.io/spack/spack/-/pipelines/176945

where mpich builds fine. That seems to suggest that a system failure was recorded as a broken spec.

[scottwittenburg]

Just pointing out that if the rebuild job can never get scheduled onto a pod, as in the jobs you linked above, then we definitely don't report anything to the broken specs list, as none of our rebuild script logic ever gets to run. In the case of the broken mpich/oxqu5m7, it appears to have been recorded on the broken specs list here: https://gitlab.spack.io/spack/spack/-/jobs/3333864. Then the other retries were already used up, or had previously failed and this was the third one.

[alalazo]

Feel free to change the title of the spec to make it more precise. What I did to track this down was:

1. Get a list of the pipelines on develop
2. Look one by one at the failed pipelines to see when the failure was happening
3. Check "failed jobs" as in the screenshot

Do I understand correctly then that "Failed jobs" might not be the job that caused the spec to be marked as "broken", but I may need to look further down in the right column, like:

[scottwittenburg]

Yes, by looking down that list at the other failures of the same hash, I found the one that actually got scheduled, ran, and the build failed (and said it was going to report the breakage).