Skip to content

Installation on ARCHER

Jump to bottom Edit New page
Phil Heron edited this page Jan 16, 2018 · 8 revisions

Author

Philip Heron and ARCHER support staff.
Email: [email protected]

Written 10th January 2018

Instructions

Setting up ASPECT on ARCHER requires a number of hours. It is best setup using submit scripts to run on the appropriate nodes. As a result, an account to a project is required for this setup.

The ARCHER support staff has made a number of submit scripts available. The zip file of the folder can be found here at https://github.com/heronphi/Archer_dealii_aspect_install. Below will go through the different files within it and how they are used.

The instructions below utilize deal.ii 8.5.1 - in order for deal.ii to be compatible with ARCHER a number of changes have been made. If a newer version of deal.ii is required, please get in contact and another setup can be obtained.

Account

As mentioned, you will need a project account to be able to run the setup scripts. In the submit scripts, the project account will be given as a A99-xacc, please replace this with your own.

Setup - deal.ii

The outline below compiles deal.ii in the work directory.

  pwd
  /work/A99/A99/username/

In your work directory, make a deal_ii folder that the build we take place.

   mkdir /work/A99/A99/username/deal_ii

Copy the contents of the Archer_dealii_aspect_install folder into deal_ii –

build.pbs			master.sh
build_p4est.sh			modules.sh
change.sh			p4est-1.1.tar.gz
configure.sh			p4est-setup.sh
dealii-8.5.1.changes		step-32_aprun_CMakeLists.txt
dealii-8.5.1.changes.tar.gz	tar.sh
dealii-8.5.1.tar.gz		test.pbs
download.sh			unpack.sh
env.sh

A master script (master.sh) has been made available to work through all of the setup required for deal.ii. The master script is given here:

. unpack.sh
. change.sh
. env.sh &> env.log
. modules.sh &> modules.log
module list >> modules.log 2>&1
. build_p4est.sh &> build_p4est.log
mkdir -p $build_dir
cd $build_dir
. ../configure.sh &> ../configure.log &&
qsub ../build.pbs
cd ..

Below I will outline what each script does, and what need to be done to ensure the build occurs in your work directory. After going through these edits, the master script should then be executed.

  1. . unpack.sh this run the unpack script, which unpacks the deal.ii 8.5.1 source code.

  2. . change.sh this runs the change script, which unpacks the changes to deal.ii 8.5.1 source code.

  3. . env.sh runs the environment script. To set up the build of deal.ii, we need to edit the env.sh build script that configures the variables of deal.ii. We need to edit install_dir line to point to the correct build directory:

export install_dir=/work/A99/A99/username/deal_ii/$version-$config_name
  1. . module.sh sets up the modules that are needed. The module list will be (including the initial modules already loaded):
Currently Loaded Modulefiles:
  1) modules/3.2.10.6
  2) eswrap/1.3.3-1.020200.1280.0
  3) switch/1.0-1.0502.60522.1.61.ari
  4) craype-network-aries
  5) craype-ivybridge
  6) craype/2.5.10
  7) pbs/12.2.401.141761
  8) cray-mpich/7.5.5
  9) packages-archer
 10) bolt/0.6
 11) nano/2.2.6
 12) leave_time/1.3.0
 13) quickstart/1.0
 14) ack/2.14
 15) xalt/0.6.0
 16) epcc-tools/6.0
 17) gcc/5.3.0
 18) cray-libsci/16.11.1
 19) udreg/2.3.2-1.0502.10518.2.17.ari
 20) ugni/6.0-1.0502.10863.8.29.ari
 21) pmi/5.0.12
 22) dmapp/7.0.1-1.0502.11080.8.76.ari
 23) gni-headers/4.0-1.0502.10859.7.8.ari
 24) xpmem/0.1-2.0502.64982.5.3.ari
 25) dvs/2.5_0.9.0-1.0502.2188.1.116.ari
 26) alps/5.2.4-2.0502.9774.31.11.ari
 27) rca/1.0.0-2.0502.60530.1.62.ari
 28) atp/2.1.0
 29) PrgEnv-gnu/5.2.82
 30) cmake/3.5.2
 31) cray-hdf5-parallel/1.10.0.1
 32) cray-netcdf-hdf5parallel/4.4.1.1
 33) cray-tpsl/17.04.1
 34) cray-trilinos/12.8.1.0

  1. . build_p4est.sh this builds the p4est library that has been downloaded into the original tar file.

  2. configure.sh the configure script for deal.ii and the ARCHER Cray Linux Environment.

7 build.pbs starts the make file in a qsub script to allow for a long build on a node. Here we need to change the account number:

#PBS -A A99-xacc
  1. Before the master script is run, we need to set the Cray linking properties to dynamic. In the terminal window, type:
export CRAYPE_LINK_TYPE=dynamic
  1. After the making the edits in points (3), (7) and (8), then master script can be executed:
. master.sh

Please note – it is important to execute this script as it is written. Please use “.[space]master.sh” and not ./master.sh, or anything else.

The unpacking, patches, and configures will take about 40 minutes. However, then the build job will then take 2 hours.

  1. We can run some tests using the test.pbs script - but the account name must be changed:
#PBS -A A99-xacc

From the build script the test script can be executed:

cd deal_ii/build
qsub ../test.pbs

However, the affinity test fails because I think it is testing the wrong thing - it compares the numbers of CPUs DEAL II is running on with the total number of CPUs * SMP threads.

Setup - ASPECT

Once deal.ii is built, we can download ASPECT and make and install.

  1. Before installing, run the modules script in the deal.ii folder:
. modules.sh

as above, this is .[space]modules.sh Check the module loaded are the same as that shown in point 4 in the deal.ii install.

module list
Currently Loaded Modulefiles:
  1) modules/3.2.10.6
  2) eswrap/1.3.3-1.020200.1280.0
  3) switch/1.0-1.0502.60522.1.61.ari
  4) craype-network-aries
  5) craype-ivybridge
  6) craype/2.5.10
  7) pbs/12.2.401.141761
  8) cray-mpich/7.5.5
  9) packages-archer
 10) bolt/0.6
 11) nano/2.2.6
 12) leave_time/1.3.0
 13) quickstart/1.0
 14) ack/2.14
 15) xalt/0.6.0
 16) epcc-tools/6.0
 17) gcc/5.3.0
 18) cray-libsci/16.11.1
 19) udreg/2.3.2-1.0502.10518.2.17.ari
 20) ugni/6.0-1.0502.10863.8.29.ari
 21) pmi/5.0.12
 22) dmapp/7.0.1-1.0502.11080.8.76.ari
 23) gni-headers/4.0-1.0502.10859.7.8.ari
 24) xpmem/0.1-2.0502.64982.5.3.ari
 25) dvs/2.5_0.9.0-1.0502.2188.1.116.ari
 26) alps/5.2.4-2.0502.9774.31.11.ari
 27) rca/1.0.0-2.0502.60530.1.62.ari
 28) atp/2.1.0
 29) PrgEnv-gnu/5.2.82
 30) cmake/3.5.2
 31) cray-hdf5-parallel/1.10.0.1
 32) cray-netcdf-hdf5parallel/4.4.1.1
 33) cray-tpsl/17.04.1
 34) cray-trilinos/12.8.1.0
  1. Before ASPECT is built, the environment needs to be set so that dynamic linking can occur:
export CRAYPE_LINK_TYPE=dynamic
  1. Wherever you would like to download ASPECT:
git clone --recursive https://github.com/geodynamics/aspect.git
  1. Enter into the ASPECT folder, and start the CMAKE by linking to the dealii build folder.
cd aspect
cmake -DDEAL_II_DIR=/work/A99/A99/username/deal_ii/build . 

-- ====================================================
-- ============ Configuring ASPECT ====================
-- ====================================================
-- Found deal.II version 8.5.1 at '/work/A99/A99/username/deal_ii/build/lib/cmake/deal.II'
-- Using ASPECT_USE_PETSC = 'OFF'
-- The C compiler identification is GNU 5.3.0
-- The CXX compiler identification is GNU 5.3.0
-- Cray Programming Environment 2.5.10 C
-- Check for working C compiler: /opt/cray/craype/2.5.10/bin/cc
-- Check for working C compiler: /opt/cray/craype/2.5.10/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.5.10 CXX
-- Check for working CXX compiler: /opt/cray/craype/2.5.10/bin/CC
-- Check for working CXX compiler: /opt/cray/craype/2.5.10/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting up test project, see tests/setup_tests.log for details.
-- Performing Test HAVE_DLOPEN
-- Performing Test HAVE_DLOPEN - Success
-- Enabling dynamic loading of plugins from the input file
-- Looking for include file link.h
-- Looking for include file link.h - found
-- Enabling checking of compatible deal.II library when loading plugins
-- Autopilot invoked
###
#
#  Project  aspect  set up with  deal.II-8.5.1  found at
#      /work/A99/A99/username/deal_ii/build
#
#  CMAKE_BUILD_TYPE:          Debug
#
#  You can now run
#       $ make                - to compile and link the program
#       $ make run            - to (compile, link and) run the program
#
#       $ make debug          - to switch the build type to 'Debug'
#       $ make release        - to switch the build type to 'Release'
#
#       $ make edit_cache     - to change (cached) configuration variables
#                               and rerun the configure and generate phases of CMake
#
#       $ make strip_comments - to strip the source files in this
#                               directory off the documentation comments
#       $ make clean          - to remove the generated executable as well as
#                               all intermediate compilation files
#       $ make runclean       - to remove all output generated by the program
#       $ make distclean      - to clean the directory from _all_ generated
#                               files (includes clean, runclean and the removal
#                               of the generated build system)
#       $ make info           - to view this message again
#
#  Have a nice day!
#
###
-- Performing Test HAVE_FP_EXCEPTIONS
-- Performing Test HAVE_FP_EXCEPTIONS - Success
-- Runtime floating point checks enabled.
-- Disabling cotire.
-- Parameter GUI not found: install and provide a hint using -D PARAMETER_GUI_DIR or set -D PARAMETER_GUI_EXECUTABLE directly.
-- Writing config into detailed.log...
-- Configuring done
-- Generating done
-- Build files have been written to: /work/A99/A99/username/aspect
  1. After this cmake, you can use the make command to generate the exectuable:
make -j<N>

(where is the number of processors you have available to run the ASPECT make). I would recommend running -j8.

  1. This will build the release executable, if the release version is needed, re-make using
make release

Running ASPECT

Running jobs on ARCHER is explained well in the ARCHER user guide. An example script is given below - it was generated using the package Bolt and is described in step 5.3 of the user guide.

The user guide gives an example as such:

"For example, to generate a job script to run an executable called 'my_prog.x' for 3 hours using 3072 parallel tasks and 12 tasks per compute node, you would use:

bolt -n 3072 -N 12 -t 3:0:0 -o my_job.bolt -j my_job -A z01-budget my_prog.x arg1 arg2

This generates the job script 'my_job.bolt' with the correct options to run 'my_prog.x' with command line arguments 'arg1' and 'arg2'. The project code against which the job will be charged is specified with the ' -A ' option."

In this instance, we want to generate a 64 CPU job for the van-keken-smooth cookbook. If we went into the cookbook folder, which is where we could like to run the van-keken-smooth file, we can generate a script:

cd cookbook 

bolt -n 64 -N 12 -t 3:0:0 -o aspect_example.bolt -j Cookbook_1 -A A99-xacc ../aspect van-keken-smooth.prm

This generates the file aspect_example.bolt

#!/bin/bash --login
#
# Parallel script produced by bolt
#        Resource: ARCHER (Cray XC30 (24-core per node))
#    Batch system: PBSPro_select
#
# bolt is written by EPCC (http://www.epcc.ed.ac.uk)
#
#PBS -l select=6
#PBS -N Cookbook_1
#PBS -A n03-durham
#PBS -l walltime=3:0:0


# Switch to current working directory
cd $PBS_O_WORKDIR

# Run the parallel program
aprun -n 64 -N 12 -S 6 -d 1  ../aspect van-keken-smooth.prm

We can execute the file using qsub:

qsub aspect_example.bolt

Help

This setup was graciously made by Mark at the Archer support staff. Any queries on the build can be made to [email protected]

Add a custom footer
Add a custom sidebar
Clone this wiki locally