Building on Summit Rhel8

The following instructions apply to the OLCF Summit system with NVIDIA V100 GPUs.

initial setup

Create a working directory to contain the build script, environment script, the source code, and build directories.

See https://docs.olcf.ornl.gov/data/project_centric.html#project-work-areas for information on possible locations.

mkdir $PROJ/pumipicDev
cd $_

environment script

Create a file env.sh with the following contents:

module load gcc/11.2.0
module load cuda/11.7.1
module load cmake/3.23.2

export root=$PWD

export OMPI_CXX=$root/kokkos/bin/nvcc_wrapper
export OMPI_CC=gcc

export build=build-dcsRhel8-gcc841
export kk=$root/${build}-kokkos/install
export engpar=$root/${build}-engpar/install
export oh=$root/${build}-omegah/install
export pp=$root/${build}-pumipic/install
export cab=$root/${build}-cabana/install
export CMAKE_PREFIX_PATH=$kk:$oh:$engpar:$pp:$cab:$CMAKE_PREFIX_PATH

build script

Create a script named buildAllSummit.sh with the following contents:

#!/bin/bash

# kokkos
cd $root
git clone -b 4.1.00 https://github.com/kokkos/kokkos.git
mkdir -p $kk
cmake -S kokkos -B ${kk%%install} \
  -DCMAKE_CXX_COMPILER=mpiCC \
  -DKokkos_ARCH_VOLTA70=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=on \
  -DCMAKE_INSTALL_PREFIX=$kk
cmake --build ${kk%%install} -j8 --target install

# Omega_h
cd $root
git clone https://github.com/SCOREC/omega_h.git
cd omega_h && git checkout e1be29b && cd -
mkdir -p $oh
cmake -S omega_h -B ${oh%%install} \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=$oh -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_CUDA=on -DOmega_h_CUDA_ARCH=70 \
  -DOmega_h_USE_Kokkos=ON -DOmega_h_USE_MPI=on \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DKokkos_PREFIX=$kk/lib64/cmake/
cmake --build ${oh%%install} -j8 --target install

#engpar
cd $root
git clone https://github.com/SCOREC/EnGPar.git
mkdir -p $engpar
cmake -S EnGPar -B ${engpar%%install} \
  -DCMAKE_INSTALL_PREFIX=$engpar \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
cmake --build ${engpar%%install} -j8 --target install

#Cabana
cd $root
git clone https://github.com/ECP-copa/Cabana.git cabana
[ -d $cab ] && rm -rf ${cab%%install}
mkdir -p $cab
cmake -S cabana -B ${cab%%install} \
  -DCMAKE_BUILD_TYPE="Release" \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DCabana_ENABLE_Cuda=ON \
  -DCMAKE_INSTALL_PREFIX=$cab
cmake --build ${cab%%install} -j8 --target install

#PUMIPic
cd $root
git clone https://github.com/SCOREC/pumi-pic.git
cd pumi-pic
sed -i -e '[email protected]:!https://github.com/!g' .gitmodules #use https
git submodule init 
git submodule update # download test data
cd -
[ -d $pp ] && rm -rf ${pp%%install}
mkdir -p $pp
cmake -S pumi-pic -B ${pp%%install} \
  -DCMAKE_CXX_COMPILER=mpicxx  \
  -DIS_TESTING=ON  \
  -DPS_IS_TESTING=ON  \
  -DTEST_DATA_DIR=$root/pumi-pic/pumipic-data  \
  -DOmega_h_PREFIX=$oh  \
  -DKokkos_PREFIX=$kk  \
  -DEnGPar_PREFIX=$engpar  \
  -DCabana_INSTALL_DIR=$cab  \
  -DENABLE_CABANA=on \
  -DCMAKE_INSTALL_PREFIX=$pp
cmake --build ${pp%%install} -j8 --target install

Make the script executable:

chmod +x buildAllSummit.sh

build

Source the environment file, and run the build script:

source env.sh
./buildAllSummit.sh