USE_CRON_TO_RELAUNCH option is broken on Cheyenne and Derecho

[mkavulich]

Expected behavior

We should be able to generate a workflow using the config option USE_CRON_TO_RELAUNCH: True

Current behavior

If the user attempts to generate the workflow with the USE_CRON_TO_RELAUNCH: True option, the workflow generation fails.

Machines affected

Cheyenne and Derecho

Steps To Reproduce

1. Check out and compile the latest code
2. Attempt to

• Example test command: ./run_WE2E_tests.py -m cheyenne -a P48503002 -t grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16 -q --launch=cron

3. See the bug:

*********************************************************************
FATAL ERROR:
Experiment generation failed. See the error message(s) printed below.
For more detailed information, check the log file from the workflow
generation script: log.run_WE2E_tests
*********************************************************************

Traceback (most recent call last):
  File "/glade/scratch/kavulich/workdir/UFS/test_develop/2023-10-06/intel/ufs-srweather-app/tests/WE2E/./run_WE2E_tests.py", line 559, in <module>
    run_we2e_tests(homedir,args)
  File "/glade/scratch/kavulich/workdir/UFS/test_develop/2023-10-06/intel/ufs-srweather-app/tests/WE2E/./run_WE2E_tests.py", line 241, in run_we2e_tests
    expt_dir = generate_FV3LAM_wflow(ushdir,logfile=f"{ushdir}/log.generate_FV3LAM_wflow",
  File "/glade/scratch/kavulich/workdir/UFS/test_develop/2023-10-06/intel/ufs-srweather-app/tests/WE2E/../../ush/generate_FV3LAM_wflow.py", line 164, in generate_FV3LAM_wflow
    add_crontab_line(called_from_cron=False,machine=expt_config["user"]["MACHINE"],
  File "/glade/scratch/kavulich/workdir/UFS/test_develop/2023-10-06/intel/ufs-srweather-app/tests/WE2E/../../ush/get_crontab_contents.py", line 89, in add_crontab_line
    crontab_cmd, crontab_contents = get_crontab_contents(called_from_cron, machine, debug)
  File "/glade/scratch/kavulich/workdir/UFS/test_develop/2023-10-06/intel/ufs-srweather-app/tests/WE2E/../../ush/get_crontab_contents.py", line 48, in get_crontab_contents
    {crontab_cmd} -l
UnboundLocalError: local variable 'crontab_cmd' referenced before assignment

Detailed Description of Fix (optional)

The variable crontab_cmd needs to be set for all cases; right now at the workflow generation step it is not set on Cheyenne or Derecho due to a logic error in the get_crontab_contents.py python script. Will open a PR shortly to fix.

[mkavulich]

Re-opening until fix is applied to release branch as well

[MichaelLueken]

This issue was completed with PR #939. Closing now.