Home

Shell script-based Regression Test: rt.sh

• Set of simple shell script files and input files
• Build + Run
• Run only (workaround)
• Supports Rocoto and ecFlow workflow managers
• Available on Hera and Orion
• ./rt.sh -k -f to run full regression test
• Regression test root directory: DATM-MOM6-CICE6/tests_datm

How to run all 4 regression test cases

• $ git clone https://github.com/ufs-community/ufs-weather-model DATM-MOM6-CICE6
• $ cd DATM-MOM6-CICE6
• $ git submodule update --init --recursive
• $ cd tests_datm
• $ vi rt.sh (The default account to use is nems. If you cannot use nems account,please change the "ACCNR" parameter, e.g., ACCNR=marine-cpu
• $ rt.sh -f -k (This command will compile the code and run all 4 test cases and keep the run directory after the runs are done.)

rt.sh related Files

• detect_machine.sh: detect and assign machine name, set account (nems is default)
• compile.sh: build a model using CMake
• Run regression test
  • run_test.sh: sets environment variables, run directory, etc., and calls rt_datm.sh
  • rt_datm.sh: prepares a canned case in the run directory, and calls rt_utils functions
  • rt_utils.sh: contains utility functions, e.g.,
    • submit_and_wait
    • check_results
    • rocoto_create_compile_task, rocoto_create_run_task, rocoto_run, rocoto_kill
    • ecflow_create_compile_task, ecflow_create_run_task, ecflow_run, ecflow_kill

Input Files: 1) rt.conf

• rt.conf: specify compile and run cases in tests directory
• COMPILE: set compiler options and machines to use
• RUN: specify <test-name> located in tests directory
• Each row is processed sequentially
• Workflow managers: RUN depends on preceding COMPILE

Input Files: 2) tests/<test-name>

• Two levels to set simulation parameters
  • default_vars.sh sets default values
  • <test-name> overrides default values, adds test-specific parameters, e.g.,
    • IATM=1760, JATM=880, FHMAX=48, WLCLK=45
• Set environment variables that are passed onto various template files in parm/
  • input.mom6.nml.IN
  • nems.configure.*.IN
  • model_configure.IN
• Specify configuration templates to use, e.g.,
  • NEMS_CONFIGURE=”nems.configure.cmeps_lstep_cold_atm_ocn_ice.IN”

Input Files: 3) datm_conf/<run-setup-name>

• Set up input data, grid data, etc. by copying files from baseline directory to run directory
• Baseline directory contains
  • Sub-directories for input data (e.g., CICE_IC, MOM6_IC, and DATM)
  • Sub-directories for previous run results (e.g., RT-Baselines_1step_cold_start_100_cmeps_CFSR)
• Make sure directories and files exist in RTPWD

Default Directories specified in rt.sh

• Baseline directory (RTPWD)
  • Hera: /scratch1/NCEPDEV/nems/emc.nemspara/RT/DATM-MOM6-CICE6/develop-YYYYMMDD
  • Orion: /work/noaa/nems/emc.nemspara/RT/DATM-MOM6-CICE6/develop-YYYYMMDD
• Run directory root (RUNDIR_ROOT)
  • Hera: RUNDIR_ROOT=/scratch1/NCEPDEV/stmp2/${USER}/S2S_RT/rt_$$
  • Orion: RUNDIR_ROOT=/work/noaa/stmp/${USER}/test/${USER}/S2S_RT/rt_$$
  • RUNDIR=${RUNDIR_ROOT}/${TEST_NAME}
• New baseline directory (NEW_BASELINE)
  • Hera: /scratch1/NCEPDEV/stmp4/${USER}/S2S_RT/REGRESSION_TEST_INTEL
  • Orion: /work/noaa/stmp/${USER}/test/${USER}/S2S_RT/REGRESSION_TEST_INTEL

Build

• Triggered by COMPILE row in rt.conf
• compile.sh is a simple wrapper around build.sh script located in DATM-MOM6-CICE6 directory.

rt.sh Usage

• ./rt.sh: display usage information
• ./rt.sh -c | -f | -l | -m | -k | -r | -e | -h
  • -c: create baseline
  • -f: use rt.conf
  • -l: use instead of rt.conf
  • -m: compare against new baseline results
  • -k: keep run directory
  • -r: use Rocoto workflow manager
  • -e: use ecFlow workflow manager
  • -h: display help (same as ./rt.sh)

Run Full Regression Tests

• If you make code changes that are not expected to change simulation results, you can run full regression tests afterward to demonstrate your changes do not break anything
• Currently, there are 4 standard regression tests on Hera and Orion
• In DATM-MOM6-CICE6/tests_datm/ directory, use any one of the following:
  • $ ./rt.sh -f >output 2>&1 &
  • $ ./rt.sh -f -e (use ecFlow)
  • $ ./rt.sh -fr (use Rocoto)
  • $ ./rt.sh -fek (use ecFlow, keep run directory for post-run diagnosis)

Run a Single Regression Test

• Create a file, say my_test.conf, with a single COMPILE and a single RUN
  • $ cp rt.conf my_test.conf
  • $ vi my_test.conf
  • $ ./rt.sh -l my_test.conf
• Or make a copy of original rt.conf file
  • $ cp rt.conf rt.conf.orig
  • $ vi rt.conf
  • $ ./rt.sh -f

Already have an Executable File

• Remove COMPILE row in rt.conf
• cp DATM-MOM6-CICE6/modulefiles/hera.intel/coupled DATM-MOM6-CICE6/tests_datm/modules.fcst (on hera)
• or cp DATM-MOM6-CICE6/modulefiles/orion.intel/coupled DATM-MOM6-CICE6/tests_datm/modules.fcst (on orion)
• $ ./rt.sh -f
• This approach does not work with workflow managers because RUN depends on COMPILE

Output Files and Log Files for Diagnosis

• Summary files
  • Hera: RegressionTests_hera.intel.log, Compile_hera.intel.log
  • Orion: RegressionTests_orion.intel.log, Compile_orion.intel.log
• ./rt.sh >output 2>&1 &: output of rt.sh
• Log files in log_hera.intel/ and log_orion.intel/
  • compile_*.log: output of compile.sh
  • run_*.log: output of run_test.sh
• Run directory RUNDIR_ROOT/
  • .log: output of rt_datm.sh. If rocoto used, also contains err & out from sbatch job
  • subdir: contains all files necessary for simulation, e.g., sbatch job_card
  • run_test*.env: exported environment variables, e.g., MACHINE_ID, RTPWD, PATHRT, ACCNR, QUEUE.