Building and Running on Frontier

The following instructions apply to the Frontier system.

initial setup

Create a working directory to contain the build script, environment script, the source code, and build directories.

environment script

Create a script env.sh with the following contents:

module reset
module load PrgEnv-amd
module load craype-accel-amd-gfx90a
module load cray-mpich
export CRAYPE_LINK_TYPE=dynamic
export MPICH_GPU_SUPPORT_ENABLED=1

build script

Create a script named installAll.sh with the following contents:

#!/bin/bash

# Kokkos
git clone -b 4.1.00 [email protected]:Kokkos/kokkos.git
bdir=$PWD/buildKokkosVega90a
rm -rf $bdir
cmake -S kokkos -B $bdir \
 -DCMAKE_BUILD_TYPE=RelWithDebInfo\
 -DCMAKE_CXX_COMPILER=CC\
 -DCMAKE_CXX_EXTENSIONS=OFF\
 -DKokkos_ENABLE_TESTS=OFF\
 -DKokkos_ENABLE_EXAMPLES=OFF\
 -DKokkos_ENABLE_SERIAL=ON\
 -DKokkos_ENABLE_OPENMP=OFF\
 -DKokkos_ENABLE_HIP=ON\
 -DKokkos_ARCH_VEGA90A=ON\
 -DKokkos_ENABLE_DEBUG=OFF\
 -DCMAKE_INSTALL_PREFIX=$bdir/install
cmake --build $bdir -j8 --target install

#Omega_h
git clone -b reducedThrust [email protected]:SCOREC/omega_h.git
bdir=$PWD/buildOmegahVega90a
rm $bdir -rf
cmake -S omega_h -B $bdir \
  -DCMAKE_INSTALL_PREFIX=$bdir/install \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_CUDA=OFF \
  -DOmega_h_USE_MPI=ON \
  -DOmega_h_USE_OpenMP=OFF \
  -DCMAKE_CXX_COMPILER=CC \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA_AWARE_MPI=ON \
  -DKokkos_PREFIX=$PWD/buildKokkosVega90a/install \
  -DBUILD_TESTING=ON
cmake --build $bdir -j8 --target install

#Engpar
git clone [email protected]:SCOREC/EnGPar.git
bdir=$PWD/buildEngparVega90a
cmake -S EnGPar -B $bdir \
  -DCMAKE_INSTALL_PREFIX=$bdir/install \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
cmake --build $bdir -j8 --target install

#Cabana
git clone https://github.com/ECP-copa/Cabana.git cabana
bdir=$PWD/buildCabanaVega90a
rm $bdir -rf
cmake -S cabana -B $bdir \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_CXX_COMPILER=CC \
  -DCMAKE_PREFIX_PATH=$PWD/buildKokkosVega90a/install \
  -DCMAKE_INSTALL_PREFIX=$bdir/install
cmake --build $bdir -j8 --target install

#Pumi-Pic
git clone --recursive -b ac/replace-cuda-thrust [email protected]:SCOREC/pumi-pic.git
bdir=$PWD/buildPumiPicVega90a
rm -rf $bdir
cmake -S pumi-pic -B $bdir \
  -DCMAKE_CXX_COMPILER=CC \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DIS_TESTING=ON \
  -DPS_IS_TESTING=ON \
  -DPS_USE_GPU_AWARE_MPI=ON \
  -DTEST_DATA_DIR=$PWD/pumi-pic/pumipic-data \
  -DOmega_h_PREFIX=$PWD/buildOmegahVega90a/install \
  -DKokkos_PREFIX=$PWD/buildKokkosVega90a/install \
  -DEnGPar_PREFIX=$PWD/buildEngparVega90a/install \
  -DCabana_PREFIX=$PWD/buildCabanaVega90a/install  \
  -DENABLE_CABANA=on \
  -DCMAKE_INSTALL_PREFIX=$bdir/install
cmake --build $bdir -j8 --target install

Make the script executable:

chmod +x installAll.sh

build

Source the environment file, and run the build script:

source env.sh
./installAll.sh

run tests

cd buildPumiPicVega90a
salloc -A PHY122 -t 00:10:00 -N 1
ctest