[release/public-v2.2.0] Reactivate comprehensive tests and fix Hera GNU pipeline issues

[MichaelLueken]

DESCRIPTION OF CHANGES:

• Changes were made to reactivate the comprehensive tests in the SRW App Jenkins pipeline via the Jenkinsfile.
• A fix was introduced to wrapper_srw_ftest.sh to correct occasional failures of the Functional Workflow Task Tests stage on Hera GNU.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

TESTS CONDUCTED:

• Jenkins
  Since this PR is only affecting the Jenkinsfile and the Jenkins scripts (correcting an issue with the Functional Workflow Task Tests and reactivating comprehensive testing), only Jenkins testing is required for this PR.

DEPENDENCIES:

None

DOCUMENTATION:

None

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes

[natalie-perlin]

Approving the changes for the release branch

[MichaelLueken]

One test failed with Hera GNU:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Peru_12km                                           COMPLETE              26.20
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         DEAD                   4.13
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS                             COMPLETE              18.37
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              45.34
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE              27.59
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              20.44
long_fcst                                                          COMPLETE              70.84
MET_verification_only_vx                                           COMPLETE               0.26
MET_ensemble_verification_only_vx_time_lag                         COMPLETE               7.96
nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              62.54
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                 283.67

The rerun of the get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 test successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200         COMPLETE              11.69
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              11.69

Two tests failed on Hera Intel:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Central_Asia_3km                                    DEAD                   5.94
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_grib2_2019061200          COMPLETE               6.30
get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2  COMPLETE             765.72
get_from_HPSS_ics_HRRR_lbcs_RAP                                    COMPLETE              14.18
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               6.49
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              13.36
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP                 COMPLETE              10.33
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2        COMPLETE               6.65
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2         COMPLETE             233.09
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16           COMPLETE             305.04
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR            COMPLETE             328.67
pregen_grid_orog_sfc_climo                                         DEAD                   3.81
----------------------------------------------------------------------------------------------------
Total                                                              DEAD                1699.58

The reruns of the custom_ESGgrid_Central_Asia_3km and pregen_grid_orog_sfc_climo tests successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_Central_Asia_3km                                    COMPLETE              25.17
pregen_grid_orog_sfc_climo                                         COMPLETE               8.29
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              33.46

Gaea C5 suffered an emergency power down earlier today, resulting in failures on that machine. Manual runs passed successfully:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community                                                          COMPLETE              43.60
custom_ESGgrid_NewZealand_3km                                      COMPLETE              47.74
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              27.22
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              31.37
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              29.86
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             310.23
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              31.30
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             274.47
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              17.22
nco_ensemble                                                       COMPLETE              99.54
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             306.62
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1219.17

Finally, the ability to manually toggle comprehensive testing in the Jenkins API has been confirmed in this PR. Moving forward with merging now.