OMPIO: Options to disable/change sharedpf MCA? #9656

ax3l · 2021-11-11T18:21:43Z

Background information

What version of Open MPI are you using?

4.1.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Via Juelich JSC syadmins (likely using Easybuild).

Please describe the system on which you are running

Operating system/version: Linux
Computer hardware: IBM - Juwels Booster
Network type: NVIDIA Mellanox HDR 200Gb/s InfiniBand

Details of the problem

We are running parallel HDF5 files on the Juelich (JSC) JUWELS Booster cluster.
With the OMPIO backend, we observe write issues of the form:

[jwb0065.juwels:28280] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/ompi.jwb0065.juwels.17674/jf.0/4086174241/openpmd_00010.h5_cid-5-28280.sm
[jwb0065.juwels:28279] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/ompi.jwb0065.juwels.17674/jf.0/4086174241/openpmd_00010.h5_cid-5-28280.sm
[jwb0065.juwels:28277] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/ompi.jwb0065.juwels.17674/jf.0/4086174241/openpmd_00010.h5_cid-5-28280.sm
[jwb0065.juwels:28278] mca_sharedfp_sm_file_open: Error, unable to open file for mmap: /tmp/ompi.jwb0065.juwels.17674/jf.0/4086174241/openpmd_00010.h5_cid-5-28280.sm
HDF5-DIAG: Error detected in HDF5 (1.10.6) MPI-process 2:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
HDF5-DIAG: Error detected in HDF5 (1.10.6) MPI-process 1:
HDF5-DIAG: Error detected in HDF5 (1.10.6) MPI-process 0:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
    major: File accessibilty
    minor: Unable to open file
HDF5-DIAG: Error detected in HDF5 (1.10.6) MPI-process 3:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1498 in H5F_open(): unable to open file: time = Thu Nov 11 18:41:15 2021
, name = 'diags/injection/openpmd_00010.h5', tent_flags = 13
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1498 in H5F_open(): unable to open file: time = Thu Nov 11 18:41:15 2021
, name = 'diags/injection/openpmd_00010.h5', tent_flags = 13

The only google result for the error message points to #4336, but that one should be fixed for the version we use (OpenMPI 4.1.1).

Following the source,
https://github.com/open-mpi/ompi/blob/v4.1.1/ompi/mca/sharedfp/sm/sharedfp_sm_file_open.c#L129-L141
the problem seems to be related to the sharedfp framework and I read in the FAQs that it has a couple of requirements on the file system capabilities, which might not be fulfilled at JUWELS booster (I am speculating as a system user here).

I was wondering if I could set options in sharedfp to modify those requirements or skip sharedfp altogether, but I could not find any. We already tried export HDF5_USE_FILE_LOCKING=FALSE on the HDF5 side, but that had no effect. Can you recommend any mca options we could try?

We also tried to using the ROMIO backend over OMPIO, but that has problems of its own on that system.

Attn & X-Ref

X-ref: ECP-WarpX/WarpX#2542 (comment)

cc @MaxThevenet @AndiH

help maybe @ggouaillardet ? :)

The text was updated successfully, but these errors were encountered:

edgargabriel · 2021-11-11T20:17:33Z

@ax3l thank you for reporting the issue. A couple of quick thoughts. OMPIO is actually able to execute without shared file pointers if the functionality is not used in the application. However, what happens here is that a sharedfp component is reporting 'I can be used', but the file open operation fails for whatever reason. Do you have any chance to check why the file in the /tmp directory could not be created?

Since the sm component is chosen, I would expect that it was a single node job, is this correct?

If we cannot fix the file open problem in /tmp, one option to circumvent things could be to disqualify all sharedfp components ( at least to make sure that this resolves all issues), e.g.

--mca sharedfp ^sm,lockedfile,individual

ax3l · 2021-11-18T17:46:44Z

Thank you so much for the feedback @edgargabriel!

Do you have any chance to check why the file in the /tmp directory could not be created?

@damianam can you comment on this? :)

Since the sm component is chosen, I would expect that it was a single node job, is this correct?

The error report we got in WarpX by @MaxThevenet was definitely a single-node job, yes:
ECP-WarpX/WarpX#2542

If we cannot fix the file open problem in /tmp, one option to circumvent things could be to disqualify all sharedfp components ( at least to make sure that this resolves all issues)

Thanks, we will try those next!
ECP-WarpX/WarpX#2542 (comment)

Another info that might help here: the problem seems to be appear when we pass GPU unified memory addresses to device global memory to MPI-I/O. This usually works fine, just raising it here to give a complete picture [ref]. From the system side, @damianam also posted now the full OpenMPI settings used on Juwels Booster.

damianam · 2021-11-19T14:30:30Z

I don't know why the file could not be created but I have a few pointers on this regard. The compute nodes are diskless, everything lives in memory. If by some chance you run too close to the limit, and then try to allocate a large file in /tmp, you are actually using some extra memory. The size we impose of /tmp is 247GB, so from that point of view it should be large enough. Permissions are also fine, as /tmp should be.

As a next step I would suggest to disable sharedfp. I guess it is possible, but I am not completely sure to be honest. I guess the syntax is OMPI_MCA="^sharedfp"?

edgargabriel · 2021-11-19T18:22:56Z

@damianam thank you for the explanation. The file that the sharedfp sm component creates is very small, just an offset value (i.e. 8 bytes, although will probably use an entire block from the file system level). I was thinking about adding something to the component query function to try to open a file itself, and if that doesn't work for whatever reason the component would disqualify itself.

damianam · 2021-11-22T16:34:58Z

That sounds like a sensible thing to do.

Besides of that, are we positive that the path to the file exists? open would fail if the path does not exist. @ax3l did you check that? Maybe we are hunting the bug in the wrong place.

hakostra · 2021-12-17T11:51:51Z

I want to mention that I have encountered the exact same issue with a different code (MGLET) running on Jureca-DC, also in Jülich SC. The MPI is OpenMPI 4.1.1. In my case I am able to mitigate the issue by setting export OMPI_MCA_sharedfp="^sm" in my jobscript.

In summary, with MGLET I have two I/O related errors with OpenMPI. The present one occurring with 'ompio' with a single node and issue #7795 when using 'romio' on several nodes...

…rations try to actually open the sharedfp/sm file during the query operation to ensure that the component can actually run. This is based on some reports on the mailing list that the sharedfp/sm operation causes problems in certain circumstances. Fixes issue open-mpi#9656 Signed-off-by: Edgar Gabriel <[email protected]>

…rations try to actually open the sharedfp/sm file during the query operation to ensure that the component can actually run. This is based on some reports on the mailing list that the sharedfp/sm operation causes problems in certain circumstances. Fixes issue open-mpi#9656 Signed-off-by: Edgar Gabriel <[email protected]> (cherry picked from commit d8464d2)

hppritcha · 2022-01-28T20:41:50Z

closed via #9788 #9787 #9786 and #9784

taken from: open-mpi/ompi#9656

ax3l mentioned this issue Nov 11, 2021

Crash in parallel simulations when flushing particle data on JUWELS Booster ECP-WarpX/WarpX#2542

Open

ax3l changed the title ~~OMPIO: Disable/Change sharedpf MCA?~~ OMPIO: Options to disable/change sharedpf MCA? Nov 11, 2021

edgargabriel mentioned this issue Dec 21, 2021

sharedfp_sm_file_component_query: add file open to ensure correct ope… #9784

Merged

hppritcha closed this as completed Jan 28, 2022

ax3l mentioned this issue Apr 4, 2022

Disable hdf5 chunking by default ComputationalRadiationPhysics/picongpu#3919

Merged

2 tasks

akohlmey added a commit to akohlmey/lammps that referenced this issue Aug 5, 2023

possible workaround for unit test failure

aac7257

taken from: open-mpi/ompi#9656

akohlmey added a commit to megmcca/lammps that referenced this issue Aug 5, 2023

possible workaround for unit test failure

003a207

taken from: open-mpi/ompi#9656

akohlmey added a commit to lammps/lammps that referenced this issue Aug 5, 2023

possible workaround for unit test failure

dbab5b6

taken from: open-mpi/ompi#9656

akohlmey added a commit to akohlmey/lammps that referenced this issue Aug 7, 2023

possible workaround for unit test failure

2987f38

taken from: open-mpi/ompi#9656

akohlmey added a commit to akohlmey/lammps that referenced this issue Aug 9, 2023

possible workaround for unit test failure

2f6dbfd

taken from: open-mpi/ompi#9656

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OMPIO: Options to disable/change sharedpf MCA? #9656

OMPIO: Options to disable/change sharedpf MCA? #9656

ax3l commented Nov 11, 2021 •

edited

Loading

edgargabriel commented Nov 11, 2021

ax3l commented Nov 18, 2021 •

edited

Loading

damianam commented Nov 19, 2021

edgargabriel commented Nov 19, 2021

damianam commented Nov 22, 2021 •

edited

Loading

hakostra commented Dec 17, 2021

hppritcha commented Jan 28, 2022

OMPIO: Options to disable/change sharedpf MCA? #9656

OMPIO: Options to disable/change sharedpf MCA? #9656

Comments

ax3l commented Nov 11, 2021 • edited Loading

Background information

What version of Open MPI are you using?

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Please describe the system on which you are running

Details of the problem

Attn & X-Ref

edgargabriel commented Nov 11, 2021

ax3l commented Nov 18, 2021 • edited Loading

damianam commented Nov 19, 2021

edgargabriel commented Nov 19, 2021

damianam commented Nov 22, 2021 • edited Loading

hakostra commented Dec 17, 2021

hppritcha commented Jan 28, 2022

ax3l commented Nov 11, 2021 •

edited

Loading

ax3l commented Nov 18, 2021 •

edited

Loading

damianam commented Nov 22, 2021 •

edited

Loading