Skip to content

Commit ca5baeb

Browse files
WalterKolczynski-NOAAWalterKolczynski-NOAA
authored
Abstract common j-job tasks (NOAA-EMC#1230)
Takes all of the tasks that are common to all j-jobs and abstracts them out into a shared script that is sourced by each job: - Set and create $DATA directory - Call setpy and set $cycle - Set pid, pgmout, and pgmerr - Source config files - Source machine environment file The common j-job header script is called by passing the job name for the `${machine}.env` files using the `-e` option, and a list of config files to source with the `-c` option. ``` ${HOMEgfs}/ush/jjob_header.sh -e [env_job] -c "[config1 [config2 [...]]]" ``` The job name argument (`-e`) is mandatory, and the config list is optional but recommend to always use as well. Some pre j-job rocoto entry scripts (`jobs/rocoto/*`) are currently doing much more than they should be. These sometimes required extra finagling, usually pre-calling the jjob header in the rocoto script before it does something. Refs: NOAA-EMC#1069
1 parent e8d4778 commit ca5baeb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+277
-2320
lines changed

Diff for: ‎env/HERA.env

+2
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,8 @@ elif [[ "${step}" = "awips" ]]; then
292292

293293
elif [[ "${step}" = "gempak" ]]; then
294294

295+
export CFP_MP="YES"
296+
295297
nth_max=$((npe_node_max / npe_node_gempak))
296298

297299
export NTHREADS_GEMPAK=${nth_gempak:-1}

Diff for: ‎env/ORION.env

+2
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,8 @@ elif [[ "${step}" = "awips" ]]; then
290290

291291
elif [[ "${step}" = "gempak" ]]; then
292292

293+
export CFP_MP="YES"
294+
293295
nth_max=$((npe_node_max / npe_node_gempak))
294296

295297
export NTHREADS_GEMPAK=${nth_gempak:-1}

Diff for: ‎jobs/JGDAS_ATMOS_ANALYSIS_DIAG

+1-43
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,7 @@
11
#! /usr/bin/env bash
22

33
source "${HOMEgfs}/ush/preamble.sh"
4-
5-
##############################################
6-
# make temp directory
7-
##############################################
8-
export DATA=${DATA:-${DATAROOT}/${jobid}}
9-
mkdir -p ${DATA}
10-
cd ${DATA}
11-
12-
13-
##############################################
14-
# Run setpdy and initialize PDY variables
15-
##############################################
16-
export cycle="t${cyc}z"
17-
setpdy.sh
18-
. ./PDY
19-
20-
21-
##############################################
22-
# Determine Job Output Name on System
23-
##############################################
24-
export pid=${pid:-$$}
25-
export pgmout="OUTPUT.${pid}"
26-
export pgmerr=errfile
27-
28-
29-
#############################
30-
# Source relevant config files
31-
#############################
32-
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
33-
configs="base anal analdiag"
34-
for config in ${configs}; do
35-
. ${EXPDIR}/config.${config}
36-
status=$?
37-
[[ ${status} -ne 0 ]] && exit ${status}
38-
done
39-
40-
41-
##########################################
42-
# Source machine runtime environment
43-
##########################################
44-
. ${HOMEgfs}/env/${machine}.env anal
45-
status=$?
46-
[[ ${status} -ne 0 ]] && exit ${status}
4+
source "${HOMEgfs}/ush/jjob_header.sh" -e "anal" -c "base anal analdiag"
475

486

497
##############################################

Diff for: ‎jobs/JGDAS_ATMOS_CHGRES_FORENKF

+1-43
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,7 @@
11
#! /usr/bin/env bash
22

33
source "${HOMEgfs}/ush/preamble.sh"
4-
5-
##############################################
6-
# make temp directory
7-
##############################################
8-
export DATA=${DATA:-${DATAROOT}/${jobid}}
9-
mkdir -p ${DATA}
10-
cd ${DATA}
11-
12-
13-
##############################################
14-
# Run setpdy and initialize PDY variables
15-
##############################################
16-
export cycle="t${cyc}z"
17-
setpdy.sh
18-
. ./PDY
19-
20-
21-
##############################################
22-
# Determine Job Output Name on System
23-
##############################################
24-
export pid=${pid:-$$}
25-
export pgmout="OUTPUT.${pid}"
26-
export pgmerr=errfile
27-
28-
29-
#############################
30-
# Source relevant config files
31-
#############################
32-
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
33-
configs="base anal echgres"
34-
for config in ${configs}; do
35-
. ${EXPDIR}/config.${config}
36-
status=$?
37-
[[ ${status} -ne 0 ]] && exit ${status}
38-
done
39-
40-
41-
##########################################
42-
# Source machine runtime environment
43-
##########################################
44-
. ${HOMEgfs}/env/${machine}.env anal
45-
status=$?
46-
[[ ${status} -ne 0 ]] && exit ${status}
4+
source "${HOMEgfs}/ush/jjob_header.sh" -e "anal" -c "base anal echgres"
475

486

497
##############################################

Diff for: ‎jobs/JGDAS_ATMOS_GEMPAK

+3-38
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,12 @@
11
#! /usr/bin/env bash
22

33
source "${HOMEgfs}/ush/preamble.sh"
4+
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak" -c "base gempak"
45

5-
##########################################################
6-
# make temp directory
7-
##########################################################
8-
export DATA=${DATA:-${DATAROOT}/${jobid}}
9-
mkdir -p ${DATA}
10-
cd ${DATA}
11-
12-
###########################################
13-
# Run setpdy and initialize PDY variables
14-
###########################################
15-
export cycle="t${cyc}z"
16-
setpdy.sh
17-
. ./PDY
18-
19-
export pid=${pid:-$$}
20-
export pgmout="OUTPUT.${pid}"
21-
export pgmerr=errfile
22-
23-
#############################
24-
# Source relevant config files
25-
#############################
26-
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
27-
configs="base gempak"
28-
for config in ${configs}; do
29-
. ${EXPDIR}/config.${config}
30-
status=$?
31-
[[ ${status} -ne 0 ]] && exit ${status}
32-
done
33-
34-
###############################################################
35-
. ${BASE_ENV}/${machine}.env gempak
36-
status=$?
37-
[[ ${status} -ne 0 ]] && exit ${status}
6+
# TODO (#1219) This j-job is not part of the rocoto suite
387

398
################################
409
# Set up the HOME directory
41-
################################
42-
export HOMEgfs=${HOMEgfs:-${PACKAGEROOT}/gfs.${gfs_ver}} # TODO: remove these as they are already defined in config.base
4310
export EXECgfs=${EXECgfs:-${HOMEgfs}/exec}
4411
export PARMgfs=${PARMgfs:-${HOMEgfs}/parm}
4512
export PARMwmo=${PARMwmo:-${HOMEgfs}/parm/wmo}
@@ -67,8 +34,6 @@ export DBNROOT=${DBNROOT:-${UTILROOT}/fakedbn}
6734
###################################
6835
# Specify NET and RUN Name and model
6936
####################################
70-
export NET=${NET:-gfs}
71-
export RUN=${RUN:-gdas}
7237
export model=${model:-gdas}
7338
export COMPONENT="atmos"
7439

@@ -83,7 +48,7 @@ if [ ${SENDCOM} = YES ] ; then
8348
fi
8449

8550

86-
# TODO: These actions belong in an ex-script not a j-job
51+
# TODO: These actions belong in an ex-script not a j-job (#1219)
8752
if [ -f ${DATA}/poescript ]; then
8853
rm ${DATA}/poescript
8954
fi

Diff for: ‎jobs/JGDAS_ATMOS_GEMPAK_META_NCDC

+3-20
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,13 @@
11
#! /usr/bin/env bash
22

3-
source "${HOMEgfs}/ush/preamble.sh"
4-
53
############################################
64
# GDAS GEMPAK META NCDC PRODUCT GENERATION
75
############################################
86

9-
##########################################################
10-
# make temp directory
11-
##########################################################
12-
export DATA=${DATA:-${DATAROOT}/${jobid}}
13-
mkdir -p ${DATA}
14-
cd ${DATA}
7+
# TODO (#1222) This j-job is not part of the rocoto
158

16-
###########################################
17-
# Run setpdy and initialize PDY variables
18-
###########################################
19-
export cycle="t${cyc}z"
20-
setpdy.sh
21-
. ./PDY
22-
23-
export pid=${pid:-$$}
24-
export pgmout="OUTPUT.${pid}"
25-
export pgmerr=errfile
9+
source "${HOMEgfs}/ush/preamble.sh"
10+
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_meta" -c "base gempak"
2611

2712
################################
2813
# Set up the HOME directory
@@ -46,8 +31,6 @@ cp ${FIXgempak}/datatype.tbl datatype.tbl
4631
###################################
4732
# Specify NET and RUN Name and model
4833
####################################
49-
export NET=${NET:-gfs}
50-
export RUN=${RUN:-gdas}
5134
export COMPONENT="atmos"
5235
export MODEL=GDAS
5336
export GRID_NAME=gdas

Diff for: ‎jobs/JGDAS_ATMOS_GLDAS

+3-45
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,11 @@
11
#! /usr/bin/env bash
22

33
source "${HOMEgfs:?}/ush/preamble.sh"
4-
5-
##############################################
6-
# Obtain unique process id (pid) and make temp directory
7-
##############################################
8-
export DATA=${DATA:-${DATAROOT}/${jobid}}
9-
mkdir -p "${DATA}"
10-
cd "${DATA}" || exit 1
11-
12-
13-
##############################################
14-
# Run setpdy and initialize PDY variables
15-
##############################################
16-
export cycle="t${cyc}z"
17-
setpdy.sh
18-
. ./PDY
19-
20-
21-
##############################################
22-
# Determine Job Output Name on System
23-
##############################################
24-
export pid=${pid:-$$}
25-
export pgmout="OUTPUT.${pid}"
26-
export pgmerr=errfile
27-
28-
29-
#############################
30-
# Source relevant config files
31-
#############################
32-
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
33-
configs="base gldas"
34-
for config in ${configs}; do
35-
. ${EXPDIR}/config.${config}
36-
status=$?
37-
[[ ${status} -ne 0 ]] && exit "${status}"
38-
done
39-
4+
source "${HOMEgfs}/ush/jjob_header.sh" -e "gldas" -c "base gldas"
405

416
if [[ "${cyc:?}" -ne "${gldas_cyc:?}" ]]; then
427
echo "GLDAS only runs for ${gldas_cyc} cycle; Skip GLDAS step for cycle ${cyc}"
8+
rm -Rf "${DATA}"
439
exit 0
4410
fi
4511

@@ -48,18 +14,10 @@ xtime=$((gldas_spinup_hours+12))
4814
if [[ "${CDATE}" -le "$(${NDATE:?} +"${xtime}" "${SDATE:?}")" ]]; then
4915
echo "GLDAS needs fluxes as forcing from cycles in previous ${xtime} hours"
5016
echo "starting from ${SDATE}. This gldas cycle is skipped"
17+
rm -Rf "${DATA}"
5118
exit 0
5219
fi
5320

54-
##########################################
55-
# Source machine runtime environment
56-
##########################################
57-
# shellcheck disable=SC1090-SC1091
58-
. "${HOMEgfs}/env/${machine:?}.env" gldas
59-
# shellcheck disable=
60-
status=$?
61-
[[ ${status} -ne 0 ]] && exit "${status}"
62-
6321

6422
##############################################
6523
# Set variables used in the exglobal script

Diff for: ‎jobs/JGDAS_ATMOS_VERFOZN

+1-25
Original file line numberDiff line numberDiff line change
@@ -4,36 +4,12 @@
44
# Set up environment for GDAS Ozone Monitor job
55
#############################################################
66
source "${HOMEgfs}/ush/preamble.sh"
7-
8-
###########################################################
9-
# make temp directories
10-
###########################################################
11-
export DATA=${DATA:-${DATAROOT}/${jobid}}
12-
mkdir -p ${DATA}
13-
cd ${DATA}
14-
15-
16-
##############################################
17-
# Run setpdy and initialize PDY variables
18-
##############################################
19-
export cycle="t${cyc}z"
20-
setpdy.sh
21-
. ./PDY
22-
23-
24-
####################################
25-
# Determine Job Output Name on System
26-
####################################
27-
export pid=$$
28-
export pgmout="OUTPUT.${pid}"
29-
export pgmerr=errfile
7+
source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
308

319

3210
###############################
3311
# Specify NET, RUN, and COMPONENT name
3412
##############################
35-
export NET=${NET:-gfs}
36-
export RUN=${RUN:-gdas}
3713
export COMPONENT="atmos"
3814

3915
export OZNMON_SUFFIX=${OZNMON_SUFFIX:-${NET}}

Diff for: ‎jobs/JGDAS_ATMOS_VERFRAD

+1-29
Original file line numberDiff line numberDiff line change
@@ -4,36 +4,8 @@
44
# Set up environment for GDAS Radiance Monitor job
55
#############################################################
66
source "${HOMEgfs}/ush/preamble.sh"
7+
source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"
78

8-
###########################################################
9-
# make temp directories
10-
###########################################################
11-
export DATA=${DATA:-${DATAROOT}/${jobid}}
12-
mkdir -p ${DATA}
13-
cd ${DATA}
14-
15-
16-
##############################################
17-
# Run setpdy and initialize PDY variables
18-
##############################################
19-
export cycle="t${cyc}z"
20-
setpdy.sh
21-
. ./PDY
22-
23-
24-
####################################
25-
# Determine Job Output Name on System
26-
####################################
27-
export pid=$$
28-
export pgmout="OUTPUT.${pid}"
29-
export pgmerr=errfile
30-
31-
32-
###############################
33-
# Specify NET, RUN, and COMPONENT name
34-
##############################
35-
export NET=${NET:-gfs}
36-
export RUN=${RUN:-gdas}
379
export COMPONENT="atmos"
3810

3911
export RAD_DATA_IN=${DATA}

0 commit comments

Comments
 (0)