Missing CICE hist files when cycle is not t00z and FHOUT_GFS_ICE is 24

[EricSinsky-NOAA]

What is wrong?

For cases where FHOUT_GFS_ICE is 24 and initialized at a time that is not 00Z, CICE files are not properly linked to COMROOT. This results in missing CICE hist files in COMROOT. After investigating this issue on WCOSS2, this happens because the raw CICE output is not averaged for a full 24 hours for the first lead time. For example, if the initialization time is 12Z and FHOUT_GFS_ICE is 24, the first lead time will only be averaged from 12Z to 00Z. Subsequent lead times are averaged from 00Z to 00Z. The global-workflow predetermines that the CICE output will be averaged from 12Z to 12z, which results in the global workflow creating links that do not match the raw CICE filenames, resulting in the CICE output not to save in COMROOT.

What should have happened?

It may be ideal that CICE be averaged for a full 24 hours for the first lead time. For example, if the initialization time is 12Z and FHOUT_GFS_ICE is 24, the first lead time should be averaged from 12Z to 12Z. Subsequent lead times should be averaged from 12Z to 12Z. An alternative positive outcome would be for the global-workflow to correctly predetermine the name of the CICE output in these particular cases.

What machines are impacted?

WCOSS2

Steps to reproduce

1. Set FHOUT_GFS_ICE to 24 in config.base.
2. Run a case that is not initialized at t00Z (e.g. t12z).

Additional information

This issue has been tested on WCOSS2, but it most likely impacts all platforms. The list of CICE output below illustrates how the global-workflow is not picking up the raw CICE output.

Do you have a proposed solution?

This issue has been initially addressed in PR #2561. In this PR, CICE_predet and CICE_postdet were modified so that the global-workflow can correctly predetermine the name of the raw CICE output (as illustrated in the image below). Also, the oceanice prod task was modified accordingly to account for the CICE filename changes in these cases. However, further fixes will need to be made to consider cases where FHMIN is not 0.

[aerorahul]

Can the output be 6 hrs, and a post-processing step be applied to do the averaging for 12hrs or 24hrs?
Since the CICE model cannot do 24hr averaging if the starting time is not 00z for the first interval, the first 24h output as labeled by the workflow is not correct.
The earliest 24h product that will be generated will be for the next period of 24hrs.
Also, the last 24h period will be absent as the end time of the forecast will not be at 00z.

[EricSinsky-NOAA]

Can the output be 6 hrs, and a post-processing step be applied to do the averaging for 12hrs or 24hrs?

Since we can assign an FHOUT specifically for CICE now using FHOUT_ICE, we can keep FHOUT_ICE set to 6 (and perhaps warn users somewhere not to try FHOUT_ICE=24). I think it is a good idea to apply a separate post-processing step to perform an average of 12 hours or 24 hours. This would remove the need for the check_ice_netcdf.sh script and we can use the standard <datadep age=120> as a dependency, which would resolve issue #2721. The question is where to put this post-processing step.

Since the CICE model cannot do 24hr averaging if the starting time is not 00z for the first interval, the first 24h output as labeled by the workflow is not correct.

A fix has been made in the workflow so that the first filename is labelled as 12h instead of 24h in cases where cycle=t12z and FHOUT_ICE=24. Please see screenshot in the proposed solution section above.

Also, the last 24h period will be absent as the end time of the forecast will not be at 00z.

That is correct. The last lead time that is generated for CICE when FHMAX=120 is averaged from 84h-108h.

[EricSinsky-NOAA]

An issue has been opened in UFS that relates to this one.

[NickSzapiro-NOAA]

May I ask if a "forecast day" history stream would work?
Like:

histfreq            = 'h'
histfreq_n          = 24
hist_avg            = .true.

rather than try to workaround calendar days.

[EricSinsky-NOAA]

May I ask if a "forecast day" history stream would work? Like:

histfreq            = 'h'
histfreq_n          = 24
hist_avg            = .true.

rather than try to workaround calendar days.

@NickSzapiro-NOAA I believe these are the settings I have been using. Here is the problematic output on WCOSS2 for a 2021032312 case where histfreq='h', histfreq_n= 24 and hist_avg= .true:

/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.149640/CICE_OUTPUT/
/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.149640/ice_in
/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.149640/ice_diag.d

[NickSzapiro-NOAA]

Thank you, @EricSinsky-NOAA . I think the problem is that 'h' is still calendar based. May I ask if ufs-community/ufs-weather-model#2437 works for you as well? Something like

histfreq            = '1'
histfreq_n          = 120
hist_avg            = .true.

where histfreq_n needs to be the number of timesteps in 24hr

[EricSinsky-NOAA]

Thank you for fixing this, @NickSzapiro-NOAA. After testing tests/parm/ice_in.IN from your ufs-weather-model branch and making the necessary adjustments to histfreq_n based on number of timesteps in 24h, I was able to get 24-hr (12Z-12Z) averaged CICE output for the 2021032312 test case. Here is the CICE output:

/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.9707/CICE_OUTPUT/
/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.9707/ice_diag.d
/lfs/h2/emc/stmp/eric.sinsky/RUNDIRS/gw_issue2721/gefs.2021032312/gefsfcst.2021032312/fcst.9707/ice_in

Some adjustments will need to be made on the global-workflow side, but it looks like the root of the problem was fixed in your ufs-weather-model branch.

[NickSzapiro-NOAA]

Good to hear! Thanks for checking!

[NickSzapiro-NOAA]

@EricSinsky-NOAA The CICE timestep stream was merged in ufs-weather-model