Update containers: el9, fedora40, noble, add arm64 for fedora40, el9, noble

[trws]

This ended up being more churn than I intended, but once I got into it here's where I ended up.

Effects:

• add containers for: ubuntu/noble, fedora/40, el9
• add
• factor out caliper build into a script, fix build on fedora and noble with 1-line patch
• factor python deps into a requirements file so we don't have to keep copy/pasting it
• change ubuntu and debian dockerfiles to use a single apt update rather than many
• add catch2 all the way at the bottom, this is mainly for sched, but if we can count on it could also be used for core testing since it helps so much with things like actually printing values of unit tests that failed, and making fixtures a whole lot easier to manage, it's also not large
• add env var to fix stupid HWLOC problem where it tries to look for opengl devices by blindly talking to a port that sometimes is an x11 server

All of these are built and pushed for all appropriate architectures. The bookworm, jammy and alpine tweaks shouldn't have any impact on existing tests, since they're additions or moving things, the rest are new containers at new names.

TODO:

• update github actions to create manifests for the new arm64 containers
• update check requirements

[trws]

Almost everything works, but there's an issue with mpi on the two most recent distros, fedora40 and noble. It makes me wonder if there's another issue with the PMI in the newer mpich maybe? The problem goes away in the test that sets PSM3_HAL (not sure what that does, or why, but it fixes it 🤷). @garlick is there a chance this is related to the PMI issue you mentioned with hydra/mpich? I remember that was going the other way, but the cause of the bootstrap issue isn't jumping out at me.

[garlick]

That was #6072 which was worked around by #6081 (merged).

Edit: meant to add that (as you alluded) this was a problem with hydra launching flux.
I don't think I tested flux launching that version of mpich.

[trws]

Right, I should have mentioned I saw that was fixed, sorry. The current behavior seems to be that "flux run -n 8 abort" say runs eight rank 0 processes. Everything works in the one test that sets PSM3_HAL though? And I can't offhand remember what that does. Clearly something going funny with PMI so I was vaguely hoping it was related, maybe needing a fix on the mpich side or a different workaround from us.

[garlick]

Looks like the PSM_HAL variable was set to address mpich problems in fedora39. Presumably we need it for fedora40 as well.

See also

• MPI tests failling in fedora 39 ci docker image #5694
• ci: add fedora39 build #5698

(Not sure I understand yet why that is needed)

[trws]

Thanks for that @garlick. I'm guessing it's related to MPI version, but it looks like what that does is turn off all multi-node features... that's potentially bad. Will do a bit more digging but this seems like an issue we may legitimately need a fix for, maybe we're picking up an MPI that needs PMIX?

[garlick]

The MPI tests do check for the "N singletons" type of error because that is a common one. For example

test_expect_success 'mpi hello various sizes' '
        run_timeout 30 flux submit --cc=1-$MAX_MPI_SIZE $OPTS \
                --watch -n{cc} ${HELLO} >hello.out &&
        for i in $(seq 1 $MAX_MPI_SIZE); do \
                grep "There are $i tasks" hello.out; \
        done
'

I think The PSM3 environment variable just needs to be set for fedora because fedora now ships an environment module with mpich (based on my comment in #5694) and presumably they included PathScale support that has to be suppressed.

I didn't look into why it's set for one fedora builder but not the other

[trws]

Have I mentioned I'm frustrated with Ubuntu lately? This is just absolutely bonkers: https://bugs.launchpad.net/ubuntu/+source/mpich/+bug/2072338

They linked their mpich with libpmix in noble, so not only do we not work with mpich, neither does hydra. 🤯

[garlick]

They linked their mpich with libpmix in noble, so not only do we not work with mpich, neither does hydra.

🤦🤦🤦🤦🤦

[grondo]

It is a bit disturbing that this change drops code coverage by >5%

codecov/project — 78.18% (-5.18%) compared to f5c5079 

[trws]

I think that might be because the el9 test just wouldn't finish. If it's still like that when it's all finishing, I'm planning to dig into it more. Maybe we're only doing coverage on an RPM distro now rather than both deb and rpm? Not sure, either way, definitely want to fix that before changing check requirements and merging this.

[grondo]

Ah, we only do two coverage runs, they take too long. Currently I think there's a a coverage build that uses the default image, then one that just runs the "system" tests.

[trws]

Ok, don't have the new codecov comment yet, but all the coverage posted and the python coverage is reporting the same numbers as on master. This should be ready for a review.

[trws]

@grondo, @garlick any further changes desired here? It would be nice to drop focal off the end before the interner pr lands in sched if that doesn't cause issues for someone, it's the only one that really doesn't support c++20 of the images we're currently supporting.

[grondo]

Let me take another look, but if all the tests are passing then it is probably all good!

It would be nice to drop focal off the end before the interner pr lands in sched if that doesn't cause issues for someone, it's the only one that really doesn't support c++20 of the images we're currently supporting.

It is probably fine to drop focal support in flux-sched whenever, nothing says that other projects have to support the same distros as flux-core. Looks like focal doesn't reach EOL until Apr 2025, though I doubt we have many users (though I still have a couple vms at that version)

[grondo]

Oh, one reminder: We should update branch protections just before merging this one (we may have to since there are names that now no longer apply.)

[grondo]

Also, it appears some subprojects use focal, it might be best to just keep that for now until they update, or we can transition them to newer images before landing this PR. (flux-coral2, flux-pmix, and flux-pam are all affected)

[trws]

Yup, various subprojects still use the older images since I started from the bottom. The theory on it being about time for focal is that it's two LTS releases ago, and the only distro we support that can't provide a reasonably recent compiler (the newest in focal repos seems to be gcc-10). The RHEL images have gone from some of the most difficult to deal with to some of the easiest, because they have been doing such a good job supporting new compilers, heck even lassen has gcc-12 on it from upstream rhel repos. That's why the older rhel isn't causing similar issues.

Anyway, I can certainly add the docker build/tag back in here while we address the downstream projects, will make that change.

[grondo]

The theory on it being about time for focal is that it's two LTS releases ago, and the only distro we support that can't provide a reasonably recent compiler (the newest in focal repos seems to be gcc-10).

Yeah, this makes sense to me!

[trws]

The focal docker build has been restored.

[grondo]

This LGTM! Just spotted one possible leftover debug line in one of the commits.

Also, had to restart the coverage build because we hit an instance of #6078

[grondo]

Leftover debug?

[trws]

Yup, thanks for that.

[grondo]

Not sure if we just want throw on a commit that sets GCOV_ERROR_FILE=/dev/null here to avoid the occasional coverage failure (#6078) (if you agree that's the right fix)

[trws]

Thanks for the pointer, I had missed that issue. Since we run tests in parallel, and some of these libraries get used in threads, I'm trying adding an explicit -fprofile-update=atomic to our coverage flags in hopes the files are getting corrupted by that rather than something more arcane. At worst it shouldn't hurt, if it doesn't take care of it we should probably do the env var as well.

[trws]

Looks like we're green here pending branch check updates.

[grondo]

I'm trying adding an explicit -fprofile-update=atomic to our coverage flags in hopes the files are getting corrupted by that rather than something more arcane

Oh cool, I searched all around and didn't find that. My only worry is that the coverage check, which already takes the longest, will be slower with atomic updates.

[grondo]

Anyway, already approved if you want to make the protected branches updates then set MWP. Thanks for the work here!

[trws]

I'm trying adding an explicit -fprofile-update=atomic to our coverage flags in hopes the files are getting corrupted by that rather than something more arcane

Oh cool, I searched all around and didn't find that. My only worry is that the coverage check, which already takes the longest, will be slower with atomic updates.

Much as I agree, I’m in favor of having the results be reliable, and would probably propose we split the coverage testing into two runners if we end up needing to. That said, the runtime after the change is actually slightly lower than in another PR currently in flight. I assume that’s just noise in the runs, but it doesn’t seem to have had a meaningful performance impact. Possibly because much of our code is already using atomic updates because it’s visible that it’s threaded at link? Not sure, either way, definitely something to keep an eye on but probably not an immediate issue.

[trws]

Ok, check requirements on master all updated, MWP set.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.37%. Comparing base (8a2d204) to head (81e233e).
Report is 536 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6128   +/-   ##
=======================================
  Coverage   83.37%   83.37%           
=======================================
  Files         521      521           
  Lines       84653    84653           
=======================================
+ Hits        70577    70582    +5     
+ Misses      14076    14071    -5     

see 10 files with indirect coverage changes