MPI_Init fails with some openMPI variants

[carlos-pinto-coelho-microsoft]

Getting the following error:
[phlrr4019:30856] mca: base: component_find: unable to open /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_sysv: /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_sysv.so: undefined symbol: opal_show_help (ignored)
[phlrr4019:30856] mca: base: component_find: unable to open /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_posix: /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_posix.so: undefined symbol: opal_shmem_base_framework (ignored)
[phlrr4019:30856] mca: base: component_find: unable to open /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_mmap: /usr/local/openmpi-1.10.3-cuda-8.0/lib/openmpi/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)

It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS

---

It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_init failed
--> Returned value Error (-1) instead of ORTE_SUCCESS

---

It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

ompi_mpi_init: ompi_rte_init failed
--> Returned "Error" (-1) instead of "Success" (0)

*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[phlrr4019:30856] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!

[carlos-pinto-coelho-microsoft]

I was searching around and found

https://svn.open-mpi.org/trac/ompi/wiki/Linkers

To debug the problem I reduced this to a simple plugin foo that I build using
/usr/local/mpi/bin/mpic++ -std=c++11 -shared foo.cpp -o foo.so -fPIC -I $TF_INC -O2
and that just calls MPI_Init inside.

Presumably this is similar to what mpi4py does but mpi4py has some dlopen “stuff” in https://bitbucket.org/mpi4py/mpi4py/src/eaf4f475857ec2330ef4781289328d1d44068460/src/dynload.c?at=master&fileviewer=file-view-default

Borrowing this
static void PyMPI_OPENMPI_dlopen_libmpi(void)
{
void handle = 0;
int mode = RTLD_NOW | RTLD_GLOBAL;
/ GNU/Linux and others */
#ifdef RTLD_NOLOAD
mode |= RTLD_NOLOAD;
#endif
if (!handle) handle = dlopen("libmpi.so.20", mode);
if (!handle) handle = dlopen("libmpi.so.12", mode);
if (!handle) handle = dlopen("libmpi.so.1", mode);
if (!handle) handle = dlopen("libmpi.so.0", mode);
if (!handle) handle = dlopen("libmpi.so", mode);
}

static int PyMPI_OPENMPI_MPI_Init(int *argc, char ***argv)
{
PyMPI_OPENMPI_dlopen_libmpi();
return MPI_Init(argc, argv);
}
#undef MPI_Init
#define MPI_Init PyMPI_OPENMPI_MPI_Init

static int PyMPI_OPENMPI_MPI_Init_thread(int *argc, char ***argv,
int required, int *provided)
{
PyMPI_OPENMPI_dlopen_libmpi();
return MPI_Init_thread(argc, argv, required, provided);
}
#undef MPI_Init_thread
#define MPI_Init_thread PyMPI_OPENMPI_MPI_Init_thread

from mpi4py "fixes" the issue for my build but perhaps it would be useful to make the two libs coexist together a bit better.

[qianglan]

@carlos-pinto-coelho-microsoft , hi , I also met this problem, I am not clearly understand your solution. First, the reason why the problem happened is because MPI_init function is not called in allreduce-test.py, am I right? so you create a new file which is foo.cpp, the file does the MPI_Init thing. After you compile the foo.cpp file to a library, how do you call foo.so in allreduce-test.cpp ?

[qianglan]

It seems the reason is that I install two version of MPI, and some environment like LD_LIBARARY_PATH , I don't change that, also it seems that I need to --disable-dlopen during the configure stage of OpenMPI,

[carlos-pinto-coelho-microsoft]

@qianglan the MPI_init in the background thread but it fails for some openmpi builds. The stuff I added above was taken from mpi4py and "fixes" the issue so I don't have to mess around with my openmpi build, which I don't even control in our cluster.

The other issue that I had with the code is that the way it calls MPI_Init in the background thread prevents this from working with other libraries that also call MPI_Init such as mpi4py. Locally, I ended up modifying the code to do the MPI_Init explicitly (or not at all for the case when I was using this and mpi4py in the same program).

[plegresl]

The typical way to write a library that uses MPI is to first call MPI_Initialized(), to check if MPI_Init() has already been called, and skip the MPI_Init() call within that library if it has already been called somewhere else.

[qianglan]

OK, so the problem happened because of twice MPI_Init execute. Thanks

[qianglan]

A simple solution is add below in allreduce-test.py file:

import ctypes
ctypes.CDLL("libmpi.so", mode=ctypes.RTLD_GLOBAL)