MPI Finializing and Caliper

[mastino]

I've encountered this issue before now, but recently run into it again after installing newer version of Caliper (2.5 and the github head). Running an MPI only application with caliper instrumentation and PAPI I get the following error once for each MPI rank:

CALIPER: (43): default: mpireport: MPI is already finalized. Cannot aggregate output.

We previously resolved this on fortran with cali_flush() calls and CALI_MPIREPORT_WRITE_ON_FINALIZE=false, but those don't help this time, at least not in my C test case.

Have the recommended MPI settings changed? I've included all my settings below if that helps.

Thanks,
Brian

export CALI_MPIREPORT_CONFIG="SELECT *,sum(papi.${metric}) GROUP BY prop:nested,mpi.rank FORMAT json"
export CALI_PAPI_COUNTERS=PAPI_TOT_CYC
export CALI_MPIREPORT_WRITE_ON_FINALIZE=false
export CALI_LOG_VERBOSITY=1

export CALI_MPIREPORT_FILENAME=test.json
mpirun -n 56 ./a.out

[daboehme]

Hi @mastino ,

By default, Caliper always tries to flush on program exit (after MPI_Finalize), which would produce this error message when using the mpireport service. You can turn that off with CALI_CHANNEL_FLUSH_ON_EXIT=false. I'd recommend always setting that when using mpireport.

The mpireport service should still produce the output if you let it flush at MPI_Finalize (which your setting disables) or you call cali_flush() in the program at some point before MPI_Finalize. Did you get the output (i.e., test.json)? If not, there's something else going on.

In a C program, the write-on-finalize option should work. That's only been an issue in Fortran, where we can't directly intercept MPI calls. You can still explicitly call cali_flush() instead of course, but that should only be necessary if you want to flush output somewhere other than at MPI_Finalize (e.g., after a loop iteration). Note that you need to enable the mpi service in addition to mpireport for it to work, though.

[mastino]

Thanks for the info! It seems to work.

What is the difference between CALI_CHANNEL_FLUSH_ON_EXIT and CALI_MPIREPORT_WRITE_ON_FINALIZE?

[daboehme]

By default, Caliper flushes data on program exit, which works for non-MPI programs or if you want to write data on each MPI rank. That's what the CALI_CHANNEL_FLUSH_ON_EXIT option controls. The mpireport service aggregates data across MPI ranks and it does so using MPI itself, so it must flush before MPI_Finalize() while MPI is still active. This is what the CALI_MPIREPORT_WRITE_ON_FINALIZE option controls.

If the CALI_CHANNEL_FLUSH_ON_EXIT option is on when using the mpireport service, it will try to flush data again at program exit (even though that already happened at MPI_Finalize), but then it finds that it can't use MPI anymore and produce the error message you saw. Hope this explains things!

[mastino]

Yes that does help; thank you again for all the support!