Building and Running on AiMOS RedHat 8

The following instructions apply to the CCI AiMOS system with NVIDIA V100 GPUs and the RedHat 8 (rhel8) operating system.

initial setup

Create a working directory to contain the build script, environment script, the source code, and build directories.

If code will be developed on CCI without frequent pushes to a remote repo, then I suggest using ~/barn. If code development will be done on another system, or code developed on CCI will be pushed frequently to a remote repo, then ~/scratch is also a valid choice.

mkdir ~/barn/pumipicDev
cd $_

environment script

Create a script envAimosRhel8.sh with the following contents:

module use /gpfs/u/software/dcs-rhel8-spack-install/v0162gccSpectrum/lmod/linux-rhel8-ppc64le/Core/
module load spectrum-mpi/10.4-2ycgnlq
module load cmake/3.20.0/1
module load netcdf-cxx4/4.3.1-gdysz4t
module load cuda/11.1-3vj4c72

export root=$PWD

export OMPI_CXX=$root/kokkos/bin/nvcc_wrapper
export OMPI_CC=gcc

export build=build-dcsRhel8-gcc841
export kk=$root/${build}-kokkos/install
export engpar=$root/${build}-engpar/install
export oh=$root/${build}-omegah/install
export pp=$root/${build}-pumipic/install
export cab=$root/${build}-cabana/install
export CMAKE_PREFIX_PATH=$kk:$oh:$engpar:$pp:$cab:$CMAKE_PREFIX_PATH

build script

Create a script named buildAllAimosRhel8.sh with the following contents:

#!/bin/bash

#kokkos
cd $root
git clone -b 3.7.01 https://github.com/kokkos/kokkos.git
mkdir -p $kk
cmake -S kokkos -B ${kk%%install} \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DKokkos_ARCH_VOLTA70=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=on \
  -DCMAKE_INSTALL_PREFIX=${kk}
cmake --build ${kk%%install} -j8 --target install

#engpar
cd $root
git clone https://github.com/SCOREC/EnGPar.git
mkdir -p $engpar
cmake -S EnGPar -B ${engpar%%install} \
  -DCMAKE_INSTALL_PREFIX=$engpar \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
cmake --build ${engpar%%install} -j8 --target install

# Omega_h
cd $root
git clone https://github.com/SCOREC/omega_h.git
mkdir -p $oh
cmake -S omega_h -B ${oh%%install} \
  -DCMAKE_INSTALL_PREFIX=$oh \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA=on \
  -DOmega_h_CUDA_ARCH=70 \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_TESTING=on  \
  -DCMAKE_CXX_COMPILER=`which mpicxx` \
  -DCMAKE_C_COMPILER=`which mpicc` \
  -DKokkos_PREFIX=$kk/lib64/cmake
cmake --build ${oh%%install} -j8 --target install

#Cabana
cd $root
git clone https://github.com/ECP-copa/Cabana.git cabana
[ -d $cab ] && rm -rf ${cab%%install}
mkdir -p $cab
cmake -S cabana -B ${cab%%install} \
  -DCMAKE_BUILD_TYPE="Release" \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DCabana_ENABLE_Cuda=ON \
  -DCMAKE_INSTALL_PREFIX=$cab
cmake --build ${cab%%install} -j8 --target install

#PUMIPic
cd $root
git clone https://github.com/SCOREC/pumi-pic.git
cd pumi-pic
sed -i -e '[email protected]:!https://github.com/!g' .gitmodules #use https
git submodule init 
git submodule update # download test data
cd -
[ -d $pp ] && rm -rf ${pp%%install}
mkdir -p $pp
cmake -S pumi-pic -B ${pp%%install} \
  -DCMAKE_CXX_COMPILER=mpicxx  \
  -DIS_TESTING=ON  \
  -DPS_IS_TESTING=ON  \
  -DTEST_DATA_DIR=$root/pumi-pic/pumipic-data  \
  -DOmega_h_PREFIX=$oh  \
  -DKokkos_PREFIX=$kk  \
  -DEnGPar_PREFIX=$engpar  \
  -DCabana_INSTALL_DIR=$cab  \
  -DENABLE_CABANA=on \
  -DCMAKE_INSTALL_PREFIX=$pp
cmake --build ${pp%%install} -j8 --target install

Make the script executable:

chmod +x buildAllAimosRhel8.sh

build

Source the environment file, and run the build script:

source envAimosRhel8.sh
./buildAllAimosRhel8.sh

running ps_combo test

Create a script named submitAimos.sh with the following content:

sbatch -N 1 -n 1 -t 10 -p el8-rpi --gres=gpu:16g:4 ./runAimos.sh

Create another script named runAimos.sh with the following content:

#!/bin/bash

module use /gpfs/u/software/dcs-rhel8-spack-install/v0162gccSpectrum/lmod/linux-rhel8-ppc64le/Core/
module load spectrum-mpi/10.4-2ycgnlq
module load cuda/11.1-3vj4c72

bin=/path/to/build-dcsRhel8-gcc841-pumipic/performance_tests/ #EDIT THIS PATH
elements=$((1024*1024))
particles=$((elements*5))
distribution=0 #even
structure=3 #dps
set -x
mpirun --bind-to core -np $SLURM_NPROCS \
  $bin/ps_combo --kokkos-map-device-id-by=mpi_rank $elements $particles $distribution $structure
set +x

Be sure to edit the path to the directory containing the ps_combo executable.

Make the scripts executable:

chmod +x submitAimos.sh runAimos.sh

Submit the job to SLURM:

./submitAimos.sh

Note, to run on the GPUs with 32GB of memory use the following script submitAimos32G.sh:

sbatch -N 1 -n 1 -t 10 --gres=gpu:32g:6 ./runAimos.sh