Description
I found that the persistent communication framework registers the progress callback of the selected component independent of whether persistent communication is actually used (https://github.com/open-mpi/ompi/blob/main/ompi/mca/part/base/part_base_select.c#L226). It's debatable whether that is necessary. However, the part/persist component (the only one we have right now) performs two atomic operations and takes a mutex independent of whether any partitioned operations have been started (https://github.com/open-mpi/ompi/blob/main/ompi/mca/part/persist/part_persist.h#L166). This is potentially costly and absolutely unnecessary. The atomic increment/decrement should at least be amended by a check for the variable's value. Or it should be removed entirely and a trylock should be used (assuming that the trylock does not issue an atomic operation if the mutex is locked). Worst of all, it affects everyone whether partitioned communication is used or not (since it's part of the progress loop by default).
I have seen about 17% of cycles being spent on the atomic increment/decrement in a single threaded execution on Summit (haven't checked other platforms but expect similar effects). It's likely worse if multiple threads end up in the progress loop.