Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release/1.5.1: update site config for Hercules (gnu+mvapich2) #849

Merged
merged 2 commits into from
Oct 30, 2023
Merged

release/1.5.1: update site config for Hercules (gnu+mvapich2) #849

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e9801d2
Switch Hercules gnu MPI library to mvapich2
climbfuji Oct 26, 2023
f458930
Update doc/source/PreConfiguredSites.rst for release 1.5.1 on Hercules
climbfuji Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions configs/sites/hercules/packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ packages:
all:
compiler:: [[email protected], [email protected]]
providers:
mpi:: [[email protected], [email protected]]
mpi:: [[email protected], [email protected]]

### MPI, Python, MKL
mpi:
Expand All @@ -13,13 +13,13 @@ packages:
prefix: /apps/spack-managed/oneapi-2023.1.0/intel-oneapi-mpi-2021.9.0-a66eaipzsnyrdgaqzxmqmqz64qzvhkse
modules:
- intel-oneapi-mpi/2021.9.0
openmpi:
mvapich2:
externals:
- spec: [email protected]%[email protected]~cuda~cxx~cxx_exceptions~java~memchecker+pmi~static~wrapper-rpath
fabrics=ucx schedulers=slurm
prefix: /work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1
- spec: [email protected]%[email protected]~cuda~debug~regcache~wrapperrpath process_managers=slurm
prefix: /work/noaa/epic/role-epic/spack-stack/hercules/mvapich2-2.3.7/gcc-11.3.1
modules:
- openmpi/4.1.5
- slurm/22.05.8
- mvapich2/2.3.7

### Modifications of common packages
# Version 2.0.8 doesn't compile on Hercules
Expand Down
35 changes: 35 additions & 0 deletions doc/modulefile_templates/mvapich2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#%Module1.0

module-whatis "Provides an mvapich2-2.3.7 installation for use with spack and gcc-13.3.1."

conflict openmpi
conflict mvapich2
conflict mpi
conflict intel-mpi
conflict intel-oneapi-mpi

proc ModulesHelp { } {
puts stderr "Provides an mvapich2-2.3.7 installation for use with spack and gcc-13.3.1."
}

if { [ module-info mode load ] && ![ is-loaded slurm/22.05.8 ] } {
module load slurm/22.05.8
}
#if { [ module-info mode load ] && ![ is-loaded ucx/1.13.1 ] } {
# module load ucx/1.13.1
#`}

# Set this value
set MPICH_PATH "/work/noaa/epic/role-epic/spack-stack/hercules/mvapich2-2.3.7/gcc-11.3.1"

prepend-path PATH "${MPICH_PATH}/bin"
prepend-path LD_LIBRARY_PATH "${MPICH_PATH}/lib"
prepend-path LIBRARY_PATH "${MPICH_PATH}/lib"
prepend-path CPATH "${MPICH_PATH}/include"
prepend-path CMAKE_PREFIX_PATH "${MPICH_PATH}"
prepend-path MANPATH "${MPICH_PATH}/share/man"

# Settings specific for Hercules
setenv MPI_ROOT ${MPICH_PATH}
setenv SLURM_MPI_TYPE "pmi2"
setenv MV2_HOMOGENEOUS_CLUSTER "1"
16 changes: 10 additions & 6 deletions doc/source/MaintainersSection.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,18 +257,22 @@ ecflow
mysql
``mysql`` must be installed separately from ``spack`` using a binary tarball provided by the MySQL community. Follow the instructions in :numref:`Section %s <MaintainersSection_MySQL>` to install ``mysql`` in ``/work/noaa/epic-ps/role-epic-ps/spack-stack/mysql-8.0.31-hercules``.

openmpi
need to load qt so to get consistent zlib (or just load zlib directly, check qt module)
mvapich2
Because of difficulties with ``openmpi`` on Hercules, we build ``mvapich2``. It is necessary to either load ``qt`` to use a consistent ``zlib``, or to load ``zlib`` directly (check the ``qt`` module). Create modulefile ``mvapich2`` from template ``doc/modulefile_templates/mvapich2``.

.. code-block:: console

module purge
module load zlib/1.2.13
module load ucx/1.13.1
./configure \
--prefix=/work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1 \
--with-ucx=$UCX_ROOT \
--with-zlib=$ZLIB_ROOT
module load slurm/22.05.8
FFLAGS=-fallow-argument-mismatch ./configure \
--prefix=/work/noaa/epic/role-epic/spack-stack/hercules/mvapich-2.3.7/gcc-11.3.1 \
--with-pmi=pmi2 \
--with-pm=slurm \
--with-slurm-include=/opt/slurm-22.05.8/include \
--with-slurm-lib=/opt/slurm-22.05.8/lib \
2>&1 | tee log.config./configure
make VERBOSE=1 -j4
make check
make install
Expand Down
16 changes: 8 additions & 8 deletions doc/source/PreConfiguredSites.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,17 @@ Directory ``configs/sites`` contains site configurations for several HPC systems
Pre-configured sites are split into two categories: Tier 1 with officially supported spack-stack installations (see :numref:`Section %s <Preconfigured_Sites_Tier1>`), and Tier 2 (sites with configuration files that were tested or contributed by others in the past, but that are not officially supported by the spack-stack team; see :numref:`Section %s <Preconfigured_Sites_Tier2>`).

=============================================================
Officially supported spack-stack 1.5.0 installations (tier 1)
Officially supported spack-stack 1.5.1 installations (tier 1)
=============================================================

Ready-to-use spack-stack 1.5.0 installations are available on the following, fully supported platforms. This version supports the JEDI Skylab release 5 of June 2023, and the UFS Weather Model of July 2023. It can also be used for testing spack-stack with other UFS applications (e.g. the UFS Short Range Weather Application, and the EMC Global Workflow). Amazon Web Services AMI are available in the US East 1 or 2 regions.
Ready-to-use spack-stack 1.5.1 installations are available on the following, fully supported platforms. This version supports the JEDI Skylab release 5 of June 2023, and the UFS Weather Model of July 2023. It can also be used for testing spack-stack with other UFS applications (e.g. the UFS Short Range Weather Application, and the EMC Global Workflow). Amazon Web Services AMI are available in the US East 1 or 2 regions.

+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| Organization | System | Compilers | Location | Maintainers |
+=====================+==================================+=================+=========================================================================================================+===============================+
| **HPC platforms** |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Hercules^* | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
| | Hercules^* | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env`` | Cam Book / Dom Heinzeller |
| MSU +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Orion | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.5.0/envs/unified-env`` | Cam Book / Dom Heinzeller |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
Expand Down Expand Up @@ -124,23 +124,23 @@ The following is required for building new spack environments and for using spac
module load ecflow/5.8.4
module load mysql/8.0.31

For ``spack-stack-1.5.0`` with Intel, load the following modules after loading mysql and ecflow:
For ``spack-stack-1.5.1`` with Intel, load the following modules after loading mysql and ecflow:

.. code-block:: console

module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
module load stack-intel/2021.9.0
module load stack-intel-oneapi-mpi/2021.9.0
module load stack-python/3.10.8
module available

For ``spack-stack-1.5.0`` with GNU, load the following modules after loading mysql and ecflow:
For ``spack-stack-1.5.1`` with GNU, load the following modules after loading mysql and ecflow:

.. code-block:: console

module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core
module load stack-gcc/11.3.1
module load stack-openmpi/4.1.5
module load stack-mvapich2/2.3.7
module load stack-python/3.10.8
module available

Expand Down