release/1.5.1: update site config for Hercules (gnu+mvapich2) (#849)

* Switch Hercules gnu MPI library to mvapich2 * Update doc/source/PreConfiguredSites.rst for release 1.5.1 on Hercules
JCSDA · Oct 30, 2023 · 03675e3 · 03675e3
1 parent d9d5c6d
commit 03675e3
Show file tree

Hide file tree

Showing 4 changed files with 59 additions and 20 deletions.
diff --git a/configs/sites/hercules/packages.yaml b/configs/sites/hercules/packages.yaml
@@ -2,7 +2,7 @@ packages:
   all:
     compiler:: [[email protected], [email protected]]
     providers:
-      mpi:: [[email protected], [email protected]]
+      mpi:: [[email protected], [email protected]]
 
 ### MPI, Python, MKL
   mpi:
@@ -13,13 +13,13 @@ packages:
       prefix: /apps/spack-managed/oneapi-2023.1.0/intel-oneapi-mpi-2021.9.0-a66eaipzsnyrdgaqzxmqmqz64qzvhkse 
       modules:
       - intel-oneapi-mpi/2021.9.0
-  openmpi:
+  mvapich2:
     externals:
-    - spec: [email protected]%[email protected]~cuda~cxx~cxx_exceptions~java~memchecker+pmi~static~wrapper-rpath
-        fabrics=ucx schedulers=slurm
-      prefix: /work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1
+    - spec: [email protected]%[email protected]~cuda~debug~regcache~wrapperrpath process_managers=slurm
+      prefix: /work/noaa/epic/role-epic/spack-stack/hercules/mvapich2-2.3.7/gcc-11.3.1
       modules:
-      - openmpi/4.1.5
+      - slurm/22.05.8
+      - mvapich2/2.3.7
 
 ### Modifications of common packages
   # Version 2.0.8 doesn't compile on Hercules

diff --git a/doc/modulefile_templates/mvapich2 b/doc/modulefile_templates/mvapich2
@@ -0,0 +1,35 @@
+#%Module1.0
+
+module-whatis "Provides an mvapich2-2.3.7 installation for use with spack and gcc-13.3.1."
+
+conflict openmpi
+conflict mvapich2
+conflict mpi
+conflict intel-mpi
+conflict intel-oneapi-mpi
+
+proc ModulesHelp { } {
+puts stderr "Provides an mvapich2-2.3.7 installation for use with spack and gcc-13.3.1."
+}
+
+if { [ module-info mode load ] && ![ is-loaded slurm/22.05.8 ] } {
+    module load slurm/22.05.8
+}
+#if { [ module-info mode load ] && ![ is-loaded ucx/1.13.1 ] } {
+#    module load ucx/1.13.1
+#`}
+
+# Set this value
+set MPICH_PATH "/work/noaa/epic/role-epic/spack-stack/hercules/mvapich2-2.3.7/gcc-11.3.1"
+
+prepend-path PATH "${MPICH_PATH}/bin"
+prepend-path LD_LIBRARY_PATH "${MPICH_PATH}/lib"
+prepend-path LIBRARY_PATH "${MPICH_PATH}/lib"
+prepend-path CPATH "${MPICH_PATH}/include"
+prepend-path CMAKE_PREFIX_PATH "${MPICH_PATH}"
+prepend-path MANPATH "${MPICH_PATH}/share/man"
+
+# Settings specific for Hercules
+setenv MPI_ROOT ${MPICH_PATH}
+setenv SLURM_MPI_TYPE "pmi2"
+setenv MV2_HOMOGENEOUS_CLUSTER "1"
diff --git a/doc/source/MaintainersSection.rst b/doc/source/MaintainersSection.rst
@@ -257,18 +257,22 @@ ecflow
 mysql
   ``mysql`` must be installed separately from ``spack`` using a binary tarball provided by the MySQL community. Follow the instructions in :numref:`Section %s <MaintainersSection_MySQL>` to install ``mysql`` in ``/work/noaa/epic-ps/role-epic-ps/spack-stack/mysql-8.0.31-hercules``.
 
-openmpi
-  need to load qt so to get consistent zlib (or just load zlib directly, check qt module)
+mvapich2
+  Because of difficulties with ``openmpi`` on Hercules, we build ``mvapich2``. It is necessary to either load ``qt`` to use a consistent ``zlib``, or to load ``zlib`` directly (check the ``qt`` module). Create modulefile ``mvapich2`` from template ``doc/modulefile_templates/mvapich2``.
 
 .. code-block:: console
 
    module purge
    module load zlib/1.2.13
    module load ucx/1.13.1
-   ./configure \
-       --prefix=/work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1  \
-       --with-ucx=$UCX_ROOT \
-       --with-zlib=$ZLIB_ROOT
+   module load slurm/22.05.8
+   FFLAGS=-fallow-argument-mismatch ./configure \
+       --prefix=/work/noaa/epic/role-epic/spack-stack/hercules/mvapich-2.3.7/gcc-11.3.1 \
+       --with-pmi=pmi2 \
+       --with-pm=slurm \
+       --with-slurm-include=/opt/slurm-22.05.8/include \
+       --with-slurm-lib=/opt/slurm-22.05.8/lib \
+       2>&1 | tee log.config./configure
    make VERBOSE=1 -j4
    make check
    make install

diff --git a/doc/source/PreConfiguredSites.rst b/doc/source/PreConfiguredSites.rst
@@ -8,17 +8,17 @@ Directory ``configs/sites`` contains site configurations for several HPC systems
 Pre-configured sites are split into two categories: Tier 1 with officially supported spack-stack installations (see :numref:`Section %s <Preconfigured_Sites_Tier1>`), and Tier 2 (sites with configuration files that were tested or contributed by others in the past, but that are not officially supported by the spack-stack team; see :numref:`Section %s <Preconfigured_Sites_Tier2>`).
 
 =============================================================
-Officially supported spack-stack 1.5.0 installations (tier 1)
+Officially supported spack-stack 1.5.1 installations (tier 1)
 =============================================================
 
-Ready-to-use spack-stack 1.5.0 installations are available on the following, fully supported platforms. This version supports the JEDI Skylab release 5 of June 2023, and the UFS Weather Model of July 2023. It can also be used for testing spack-stack with other UFS applications (e.g. the UFS Short Range Weather Application, and the EMC Global Workflow). Amazon Web Services AMI are available in the US East 1 or 2 regions.
+Ready-to-use spack-stack 1.5.1 installations are available on the following, fully supported platforms. This version supports the JEDI Skylab release 5 of June 2023, and the UFS Weather Model of July 2023. It can also be used for testing spack-stack with other UFS applications (e.g. the UFS Short Range Weather Application, and the EMC Global Workflow). Amazon Web Services AMI are available in the US East 1 or 2 regions.
 
 +---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 | Organization        | System                           | Compilers       | Location                                                                                                | Maintainers                   |
 +=====================+==================================+=================+=========================================================================================================+===============================+
 | **HPC platforms**                                                                                                                                                                                                  |
 +---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
-|                     | Hercules^*                       | GCC, Intel      | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env``                   | Cam Book / Dom Heinzeller     |
+|                     | Hercules^*                       | GCC, Intel      | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env``                   | Cam Book / Dom Heinzeller     |
 | MSU                 +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
 |                     | Orion                            | GCC, Intel      | ``/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.5.0/envs/unified-env``                      | Cam Book / Dom Heinzeller     |
 +---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
@@ -124,23 +124,23 @@ The following is required for building new spack environments and for using spac
    module load ecflow/5.8.4
    module load mysql/8.0.31
 
-For ``spack-stack-1.5.0`` with Intel, load the following modules after loading mysql and ecflow:
+For ``spack-stack-1.5.1`` with Intel, load the following modules after loading mysql and ecflow:
 
 .. code-block:: console
 
-   module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core
+   module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
    module load stack-intel/2021.9.0
    module load stack-intel-oneapi-mpi/2021.9.0
    module load stack-python/3.10.8
    module available
 
-For ``spack-stack-1.5.0`` with GNU, load the following modules after loading mysql and ecflow:
+For ``spack-stack-1.5.1`` with GNU, load the following modules after loading mysql and ecflow:
 
 .. code-block:: console
 
-   module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core
+   module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
    module load stack-gcc/11.3.1
-   module load stack-openmpi/4.1.5
+   module load stack-mvapich2/2.3.7
    module load stack-python/3.10.8
    module available