[develop] Fixing several issues, including 966 (bash octal issue); ad…

…d new winter weather verification test with staged data (ufs-community#997)

New test
* The new test MET_ensemble_verification_winter_wx is added. This test will exercise a number of yet-untested capabilities in the workflow, including a 10-member ensemble, snowfall verification with staged data (so can be run on all platforms, not just Jet and Hera), and several SPP settings.
* As part of this new test, snowfall observations will now be staged on all tier-1 platforms, as well as netCDF GFS data and other observation types, all for the date 2022020300

Resolved issues
* Incorrect octal notation causing ensemble vx to fail ufs-community#966 Resolved: In several locations, an explicit conversion is done to ENSMEM_INDX to ensure it is a base-10 integer, to avoid problems with bash interpreting numbers with a leading zero as octal.
* Should "EXPT_SUBDIR" be a mandatory variable? ufs-community#978 resolved: per discussion in a recent SRW code management meeting, give EXPT_SUBDIR a default value "experiment" to avoid unnecessary complications and work for users. Additionally, the default behavior if an experiment directory already exists is changed to "quit" rather than "delete"
* Issue mentioned in this discussion; the setting fhzero=6 is removed from the weather model namelist for CCPP suite FV3_GFS_v17_p8, which allows precipitation and other accumulations to be made every hour rather than 6 hours (SRW output is always hourly, so this makes sense). Also, update diag_table.FV3_GFS_v17_p8 so that all output files will be hourly
* Per discussion in [release/public-v2.2.0] Fix crontab bug for Cheyenne and Derecho, update PR template for new platforms ufs-community#939, remove an unnecessary special case in get_crontab_contents.py for Derecho

Other fixes
* Some old files for unsupported/removed CCPP suites are removed
* Add some missing task dependencies for retrieving verification obs

General improvements
* Many improvements to verification obs-pulling task
   * NDAS observations are now retrieved for forecast hour zero, and a better obs file is retrieved for major obs times (00z, 06z, 12z, 18z) per EMC guidance
   * Better in-line comments/documentation
   * Standardize order and messaging for file-on-disk checks across all observation types
* Added explanatory comments for reflectivity field in diag_table files
* Update diag_table.FV3_GFS_v17_p8 so that all output files will be hourly
* Simplify task dependencies that rely on staged verification observations; these "get_obs" tasks should always be run (they check that the data exists before trying to retrieve it), so no need to make the dependency conditional
* Add a check in monitor_jobs.py to ensure the yaml file does not contain duplicate experiment directories
* Make sure the key in the experiment dictionary used by is unique by appending the current date/time to the exptdir name; additionally, set this key as the WORKFLOW_ID variable (so that it could be used in the workflow if necessary).

parm/FV3.input.yml

@@ -524,7 +524,6 @@ FV3_GFS_v17_p8:
     doGP_cldoptics_LUT: False
     doGP_lwscat: False
     dt_inner: 150
-    fhzero: 6
     frac_grid: True
     gwd_opt: 2
     hybedmf: False

parm/diag_table.FV3_GFS_v15_thompson_mynn_lam3km

@@ -145,6 +145,7 @@
 "gfs_phys",    "t02min",     "t02min",    "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "rh02max",    "rh02max",    "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "rh02min",    "rh02min",    "fv3_history2d",  "all",  .false.,  "none",  2
+# Reflectivity from Thompson microphysics
 "gfs_phys",    "refl_10cm",  "refl_10cm",  "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "cldfra",     "cldfra",     "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "pratemax",   "pratemax",   "fv3_history2d",  "all",  .false.,  "none",  2

parm/diag_table.FV3_GFS_v15p2

@@ -277,6 +277,8 @@
 "gfs_phys",    "nbdsf_ave",    "nbdsf_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "nddsf_ave",    "nddsf_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "trans_ave",    "trans_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
+# Reflectivity from GFDL microphysics
+"gfs_phys",    "refl_10cm",    "refl_10cm",     "fv3_history2d",  "all",  .false.,  "none",  2
 # Stochastic physics
 "gfs_phys",    "sppt_wts",      "sppt_wts",      "fv3_history",  "all", .false., "none", 2
 "gfs_phys",    "skebu_wts",     "skebu_wts",     "fv3_history",  "all", .false., "none", 2

parm/diag_table.FV3_GFS_v16

@@ -277,6 +277,8 @@
 "gfs_phys",    "nbdsf_ave",    "nbdsf_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "nddsf_ave",    "nddsf_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "trans_ave",    "trans_ave",     "fv3_history2d",  "all",  .false.,  "none",  2
+# Reflectivity from GFDL microphysics
+"gfs_phys",    "refl_10cm",    "refl_10cm",     "fv3_history2d",  "all",  .false.,  "none",  2
 # Stochastic physics
 "gfs_phys",    "sppt_wts",      "sppt_wts",      "fv3_history",  "all", .false., "none", 2
 "gfs_phys",    "skebu_wts",     "skebu_wts",     "fv3_history",  "all", .false., "none", 2

parm/diag_table.FV3_GFS_v17_p8

@@ -4,8 +4,8 @@
 #output files
 "grid_spec",              -1,  "months",   1, "days",  "time"
 "atmos_static",           -1,  "hours",    1, "hours", "time"
-"fv3_history",             3,  "hours",    1, "hours", "time"
-"fv3_history2d",           3,  "hours",    1, "hours", "time"
+"fv3_history",             0,  "hours",    1, "hours", "time"
+"fv3_history2d",           0,  "hours",    1, "hours", "time"
 
 #
 #=======================
@@ -154,6 +154,8 @@
 "gfs_phys",  "BC_AOD_550",    "bc_aod550",     "fv3_history2d",      "all",  .false.,  "none",  2
 "gfs_phys",  "OC_AOD_550",    "oc_aod550",     "fv3_history2d",      "all",  .false.,  "none",  2
 "gfs_phys",  "SS_AOD_550",    "ss_aod550",     "fv3_history2d",      "all",  .false.,  "none",  2
+# Reflectivity from Thompson microphysics
+"gfs_phys",  "refl_10cm",     "refl_10cm",     "fv3_history2d",      "all",  .false.,  "none",  2
 
 "gfs_sfc",     "crain",       "crain",         "fv3_history2d",      "all",  .false.,  "none",  2
 "gfs_sfc",     "tprcp",       "tprcp",         "fv3_history2d",      "all",  .false.,  "none",  2

parm/diag_table.FV3_HRRR

@@ -117,6 +117,7 @@
 "gfs_dyn",     "graupel",     "grle",      "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "ps",          "pressfc",   "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "hs",          "hgtsfc",    "fv3_history",    "all",  .false.,  "none",  2
+# Reflectivity from Thompson microphysics
 "gfs_phys",    "refl_10cm"    "refl_10cm"  "fv3_history",    "all",  .false.,  "none",  2
 "gfs_phys",    "cldfra",      "cldfra",    "fv3_history",    "all",  .false.,  "none",  2
 #"gfs_dyn",     "ice_nc",      "nicp",      "fv3_history",  "all",  .false.,  "none",  2

parm/diag_table.FV3_RAP

@@ -161,7 +161,8 @@
 "gfs_phys",  "v-gwd_ave",     "v-gwd_ave",   "fv3_history2d",         "all",  .false.,  "none",  2
 "gfs_phys",  "dusfc",         "uflx_ave",    "fv3_history2d",         "all",  .false.,  "none",  2
 "gfs_phys",  "dvsfc",         "vflx_ave",    "fv3_history2d",         "all",  .false.,  "none",  2
-#"gfs_phys",  "refl_10cm",     "refl10cm",    "fv3_history2d",         "all",  .false.,  "none",  2
+# Reflectivity from Thompson microphysics
+"gfs_phys",  "refl_10cm",     "refl_10cm",    "fv3_history2d",         "all",  .false.,  "none",  2
 
 "gfs_phys",    "psurf",       "pressfc",   "fv3_history2d",  "all",  .false.,  "none",  2
 "gfs_phys",    "u10m",        "ugrd10m",   "fv3_history2d",  "all",  .false.,  "none",  2

parm/diag_table.FV3_RRFS_v1beta

@@ -117,6 +117,7 @@
 "gfs_dyn",     "graupel",     "grle",      "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "ps",          "pressfc",   "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "hs",          "hgtsfc",    "fv3_history",    "all",  .false.,  "none",  2
+# Reflectivity from Thompson microphysics
 "gfs_phys",    "refl_10cm"    "refl_10cm"  "fv3_history",    "all",  .false.,  "none",  2
 "gfs_phys",    "cldfra",      "cldfra",    "fv3_history",    "all",  .false.,  "none",  2
 #"gfs_dyn",     "ice_nc",      "nicp",      "fv3_history",  "all",  .false.,  "none",  2

parm/diag_table.FV3_WoFS_v0

@@ -125,6 +125,7 @@
 "gfs_dyn",     "ccn_nc",      "ccn",       "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "ps",          "pressfc",   "fv3_history",    "all",  .false.,  "none",  2
 "gfs_dyn",     "hs",          "hgtsfc",    "fv3_history",    "all",  .false.,  "none",  2
+# Reflectivity from NSSL microphysics
 "gfs_phys",    "refl_10cm"    "refl_10cm"  "fv3_history",    "all",  .false.,  "none",  2
 #"gfs_phys",    "cldfra",      "cldfra",    "fv3_history",    "all",  .false.,  "none",  2
 

parm/wflow/verify_det.yaml

@@ -73,6 +73,10 @@ metatask_GridStat_NOHRSC_all_accums_all_mems:
         SLASH_ENSMEM_SUBDIR_OR_NULL: '{% if global.DO_ENSEMBLE %}{{ "/mem#mem#" }}{% endif %}'
       walltime: 02:00:00
       dependency:
+        and:
+          taskdep:
+            attrs:
+              task: get_obs_mrms
           taskdep_pcpcombine_fcst:
             attrs:
               task: run_MET_PcpCombine_fcst_ASNOW#ACCUM_HH#h_mem#mem#
@@ -98,14 +102,9 @@ metatask_GridStat_MRMS_all_mems:
       walltime: 02:00:00
       dependency:
         and:
-          or_get_obs: # Ensure get_obs task is complete if it's turned on
-            not:
-              taskvalid:
-                attrs:
-                  task: get_obs_mrms
-            taskdep:
-              attrs:
-                task: get_obs_mrms
+          taskdep:
+            attrs:
+              task: get_obs_mrms
           datadep_post_files_exist:
             attrs:
               age: 00:00:00:30

parm/wflow/verify_ens.yaml

@@ -82,9 +82,13 @@ metatask_GenEnsProd_EnsembleStat_NOHRSC:
       <<: *envars_GenEnsProd_NOHRSC
       MET_TOOL: 'ENSEMBLESTAT'
     dependency:
-      taskdep_genensprod:
-        attrs:
-          task: run_MET_GenEnsProd_vx_ASNOW#ACCUM_HH#h
+      and:
+        taskdep:
+          attrs:
+            task: get_obs_nohrsc
+        taskdep_genensprod:
+          attrs:
+            task: run_MET_GenEnsProd_vx_ASNOW#ACCUM_HH#h
 
 metatask_GenEnsProd_EnsembleStat_MRMS:
   var:
@@ -101,14 +105,9 @@ metatask_GenEnsProd_EnsembleStat_MRMS:
       OBTYPE: 'MRMS'
     dependency:
       and:
-        or_get_obs: # Ensure get_obs task is complete if it's turned on
-          not:
-            taskvalid:
-              attrs:
-                task: get_obs_mrms
-          taskdep:
-            attrs:
-              task: get_obs_mrms
+        taskdep:
+          attrs:
+            task: get_obs_mrms
         metataskdep_post_files_exist: &post_files_exist
           attrs:
             metatask: check_post_output_all_mems

parm/wflow/verify_pre.yaml

@@ -115,7 +115,6 @@ metatask_PcpCombine_obs:
       OBTYPE: CCPA
       OBS_DIR: '&CCPA_OBS_DIR;'
       MET_TOOL: 'PCPCOMBINE'
-      ENSMEM_INDX: ''
     dependency:
       and:
         datadep:

scripts/exregional_check_post_output.sh

@@ -82,7 +82,7 @@ time_lag=$( bc -l <<< "${ENS_TIME_LAG_HRS[$i]}*${SECS_PER_HOUR}" )
 #
 #-----------------------------------------------------------------------
 #
-ensmem_indx=$(printf "%0${VX_NDIGITS_ENSMEM_NAMES}d" "${ENSMEM_INDX}")
+ensmem_indx=$(printf "%0${VX_NDIGITS_ENSMEM_NAMES}d" $(( 10#${ENSMEM_INDX})))
 ensmem_name="mem${ensmem_indx}"
 FCST_INPUT_FN_TEMPLATE=$( eval echo ${FCST_SUBDIR_TEMPLATE:+${FCST_SUBDIR_TEMPLATE}/}${FCST_FN_TEMPLATE} )
 set_vx_fhr_list \