cpld_gridgen - Update regression tests to run in parallel

[GeorgeGayno-NOAA]

DESCRIPTION OF CHANGES:

Currently, each regression test runs in sequence. These updates submit all tests to the batch queue at once, where they run independently. Then, when they are finished, a 'summary' job is triggered to create some final log files, check for errors and create a "summary" log. All other functions, such as options to build the executable (rt.sh -b), create a new baseline (rt.sh -c) and test against the new baseline (rt.sh -m) are retained.

Some specifics:

• The check_results function was moved from rt.sh to ./ush/cpld_gridgen.sh
• New script rt_summary.sh is called from rt.sh. It checks for errors, concatenates the log files from the individual tests to a final log file, checks for errors and creates a summary file.
• Removed some now unnecessary exports from the scripts.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

• Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules and WCOSS2). Done using 878af1c
• Compile branch on Hera using GNU. Done using 9c15d3a.
• Compile branch in 'Debug' mode on WCOSS2. Done using 9c15d3a.
• Compile with Doxygen on any machine with no errors. Done on Hera using 9c15d3a.
• Run unit tests locally on any Tier 1 machine. Done on Hera using 9c15d3a. All tests passed.
• Run cpld_gridgen consistency tests locally on all Tier 1 machines. Done use bd68986
• Run cpld_gridgen consistency tests using the -bc option on any Tier 1 machine to test the build executable option and to create a new baseline. Done on Hercules using db68986.
• Run cpld_gridgen consistency tests using the -m option on any Tier 1 machine to compare against the new baseline created by the above test. Done on Hercules using db68986.

Additional tests:

• Remove the executable and see if script exits with error message: cpld_gridgen - Update regression tests to run in parallel #1027 (comment)
• Invoke ./rt.sh -h to ensure the help menu is displayed: cpld_gridgen - Update regression tests to run in parallel #1027 (comment)

DEPENDENCIES:

None.

DOCUMENTATION:

No Doxygen or RST file updated required.

ISSUE:

Fixes #1016.

[GeorgeGayno-NOAA]

@DeniseWorthen - I would like you to review this. No hurry. The overall functionality is unchanged.

[GeorgeGayno-NOAA]

I still have some final cleanup to do.

[DeniseWorthen]

Oh that's great! I'll take a look when I've got a chance.

[GeorgeGayno-NOAA]

Besides the standard tests listed in the description, other tests were performed to ensure that all functionality is retained.

Using 9c15d3a on Hercules, the executable directory was removed. When rt.sh was invoked, the correct error message was display and processing stopped:

cpld_gridgen exe file is not found in /work2/noaa/da/ggayno/save/UFS_UTILS/exe/. Try -b to build or -h for help.

[GeorgeGayno-NOAA]

Using 9c15d3a, when ./rt.sh is invoked with the -h option, the correct message is displayed:

Usage: rt.sh [-c] [-m] [-h] [-b]

  -b build the executable

  -c create a new baseline

  -m compare against the new baseline

  -h display this help and exit

  Examples

    './rt.sh -b'  build exe file. compare against the existing baseline
    './rt.sh -bc' build exe file. create a new baseline
    './rt.sh -m'  do not build exe file. compare against the new baseline

[GeorgeGayno-NOAA]

@DeniseWorthen - now it is ready for review.

[DeniseWorthen]

I ran the cpld_gridgen RTs to try out this feature.

I noticed that the output no longer gets directed to a specific rt_jobnum directory. For example, on hera previously the test would go into /scratch2/NCEPDEV/stmp1/Denise.Worthen/CPLD_GRIDGEN/rt_xxxxxx. Now everything just goes into the CPLD_GRIDGEN directory. Is that how it is supposed to work?

[GeorgeGayno-NOAA]

I ran the cpld_gridgen RTs to try out this feature.

I noticed that the output no longer gets directed to a specific rt_jobnum directory. For example, on hera previously the test would go into /scratch2/NCEPDEV/stmp1/Denise.Worthen/CPLD_GRIDGEN/rt_xxxxxx. Now everything just goes into the CPLD_GRIDGEN directory. Is that how it is supposed to work?

When testing, I would get confused which directory to look in. So, I removed it. I am happy to add it back if you want.

[DeniseWorthen]

Yes please. I do use the feature that running the rt produces a unique directory.

[DeniseWorthen]

I hope to look at this by the end of today or Monday at the latest. Apologies for the delay.

[GeorgeGayno-NOAA]

I hope to look at this by the end of today or Monday at the latest. Apologies for the delay.

No hurry.

[DeniseWorthen]

@GeorgeGayno-NOAA I've tested this on both Hercules and Hera and it works as expected. I also forced a test failure, and that also worked as expected. Thanks for making this change.

[GeorgeGayno-NOAA]

@GeorgeGayno-NOAA I've tested this on both Hercules and Hera and it works as expected. I also forced a test failure, and that also worked as expected. Thanks for making this change.

Thanks, Denise. I am making a similar change to the ocnice_prep code (#1029).