Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GW-RT global_control.nml modifications #2425

Merged
merged 31 commits into from
Oct 31, 2024

Conversation

dpsarmie
Copy link
Collaborator

@dpsarmie dpsarmie commented Sep 3, 2024

Commit Queue Requirements:

  • Fill out all sections of this template.
  • All sub component pull requests have been reviewed by their code managers.
  • Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
  • Commit 'test_changes.list' from previous step

There is now an option to allow default_vars.sh to set variables that are dependent on DT_ATMOS and ATMRES. If you wish to use a non-default value for either variable and have the script set all dependent variables automatically (ex. DT_INNER, CDMBGWD, namsfc filenames, etc.), then DT_ATMOS and ATMRES must be defined before calling the export functions.

If the user does not define the variable, then the defaults of C96 (ATMRES) and 1800 (DT_ATMOS for atm only) or 720 (DT_ATMOS for cpld cases) will be set.

Example:
Your RT uses C48 resolution and you want a timestep of 720:
In your test configuration:

export ATMRES=C48
export DT_ATMOS=720
export_fv3
export_ugwpv1
export_cpl
<the rest of the config>

This will now be able to use the user defined timestep and resolution to populate the other variables needed to run this case. As always, the order of the export function calls should go export_fv3, export_ugwpv1, then export_cpl. You can remove export function calls as needed for your particular case.

You are still able to use the old methodology of defining all variables manually. This feature just allows users to streamline that process.

Non-standard RT configurations

It should be noted that many of the RTs have changes to variables from what is being set in default_vars.sh (mainly N_SPLIT, K_SPLIT, and DT_ATMOS). This was done because the runtime of the default settings was too long for the regression test suite. The RTs are made to provide a fidelity check on future UFS updates, so these checks are still valid even if some of the variables that affect runtime have been changed.
The runs using the default values were checked for stability and passed; however, there are no plans to regularly check the RTs using the default values on a regular basis.

Description:

This PR will modify the global_control.nml to add variables that are present in the Global Workflow scripts but not currently present in the RTs. Some variables will also be changed from constants to variable placeholders. This will allow Global Workflow to use the global_control.nml as a template for their operations.
Variable values that were hard-coded in global_control.nml.IN were replaced with an @ tag and the values were moved to default_vars.sh.

This PR is now also going to include a fix for #2443. This will add baseline changes that are unrelated to the initial global_control.nml modifications or the sfs configuration fix.
Default values for DT_ATMOS and ATMRES are now being set as intended. If the user defines either of these variables before the export_ function calls, then the appropriate dependent variables will be auto-configured. This removes the need to redefine these variables in the test configs.

This PR now includes a feature mentioned in #2470. The checks have been added to run_test.sh.
Changes in run_test.sh now have checks for the fast and slow coupling frequencies and a check on the CICE to ATM timesteps.

Commit Message:

* UFSWM - Modify global_control.nml to add compatibility with Global Workflow

Priority:

  • Normal

Git Tracking

UFSWM:

Sub component Pull Requests:

  • None

UFSWM Blocking Dependencies:

  • None

Changes

Regression Test Changes (Please commit test_changes.list):

  • cpld_control_p8_mixedmode intel
  • cpld_control_gfsv17 intel
  • cpld_control_gfsv17_iau intel
  • cpld_restart_gfsv17 intel
  • cpld_mpi_gfsv17 intel
  • cpld_control_sfs intel
  • cpld_debug_gfsv17 intel
  • cpld_control_p8 intel
  • cpld_control_p8.v2.sfc intel
  • cpld_restart_p8 intel
  • cpld_control_qr_p8 intel
  • cpld_restart_qr_p8 intel
  • cpld_2threads_p8 intel
  • cpld_decomp_p8 intel
  • cpld_mpi_p8 intel
  • cpld_control_ciceC_p8 intel
  • cpld_control_c192_p8 intel
  • cpld_restart_c192_p8 intel
  • cpld_bmark_p8 intel
  • cpld_restart_bmark_p8 intel
  • cpld_s2sa_p8 intel
  • cpld_control_noaero_p8 intel
  • cpld_control_nowave_noaero_p8 intel
  • cpld_debug_p8 intel
  • cpld_debug_noaero_p8 intel
  • cpld_control_c48 intel
  • cpld_warmstart_c48 intel
  • cpld_restart_c48 intel
  • cpld_control_p8_faster intel
  • cpld_control_pdlib_p8 intel
  • cpld_restart_pdlib_p8 intel
  • cpld_mpi_pdlib_p8 intel
  • cpld_debug_pdlib_p8 intel
  • atm_ds2s_docn_dice intel
  • cpld_control_nowave_noaero_p8 gnu
  • cpld_control_pdlib_p8 gnu
  • cpld_debug_pdlib_p8 gnu

Input data Changes:

  • None.

Library Changes/Upgrades:

  • No Updates

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Derecho
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
  • opnReqTest (complete task if unnecessary)

dpsarmie and others added 2 commits August 30, 2024 23:49
This adds new options to the namelist that are used and
dynamic in the global workflow. This is to prepare the RT namelist
for the RT-GW bridge PR.
@dpsarmie dpsarmie marked this pull request as ready for review September 6, 2024 18:01
This commit changes a couple of the parameters in
cpld_control_sfs to bring it in line with the Global
Workflow parameter values.
@dpsarmie dpsarmie added the Baseline Updates Current baselines will be updated. label Sep 16, 2024
@dpsarmie dpsarmie marked this pull request as draft September 17, 2024 15:49
dpsarmie and others added 2 commits September 20, 2024 08:07
This commit addresses the issue in default_vars.sh where
ATMRES is reset in the export_cpl() call to C96.
Default values will now only be set if the resolutions have
not been previously set.
@NickSzapiro-NOAA
Copy link
Collaborator

@dpsarmie I also hit a problem with DT_ATMOS. For gfsv17 tests for example:

  1. export_fv3 sets a default DT_ATMOS
  2. export_cpl sets a different default DT_ATMOS that is used in export_{cice,cmeps,...} for DT_CICE, coupling_interval_fast_sec,..
  3. Then export_ugwpv1 sets a different DT_ATMOS based on ATMRES
  4. But cice, cmeps,... never see changes from step 3

Maybe export_cpl should not touch DT_ATMOS either.

It may also be better in these tests to call export_fv3, then export_ugwpv1, then export_cpl so atmosphere is configured properly before setting other components

@dpsarmie dpsarmie marked this pull request as ready for review September 21, 2024 23:45
@dpsarmie
Copy link
Collaborator Author

dpsarmie commented Sep 23, 2024

Thanks @NickSzapiro-NOAA for bringing that up. I'll have to do some more testing to see what makes sense. I think the simplest solution would be to make the atm only and cpld default DT_ATMOS the same, but I want to make sure that the runtime isn't greatly affected for the atm only cases.
If not, then there needs to be a straightforward way to differentiate between the possible combinations and what needs to be defined.
Although I agree that the ugwpv1 call should come before the cpld call to resolve your issue 4.

@dpsarmie dpsarmie marked this pull request as draft September 23, 2024 13:38
@NickSzapiro-NOAA
Copy link
Collaborator

I think it is better if DT_ATMOS is thought of as input to export_cpl (and not modified in export_cpl). The issue I'm focusing on is (coupling) timesteps for gfsv17 test, where ufs.configure is the same before and after 2363:

# CMEPS warm run sequence
runSeq::
@3600
   MED med_phases_prep_wav_avg
   MED med_phases_prep_ocn_avg
   MED -> WAV :remapMethod=redist
   MED -> OCN :remapMethod=redist
   WAV
   OCN
   @720
     MED med_phases_prep_atm
     MED med_phases_prep_ice
     MED -> ATM :remapMethod=redist
     MED -> ICE :remapMethod=redist
     ATM
     ICE
     ATM -> MED :remapMethod=redist
     MED med_phases_post_atm
     ICE -> MED :remapMethod=redist
     MED med_phases_post_ice
     MED med_phases_ocnalb_run
     MED med_phases_prep_ocn_accum
     MED med_phases_prep_wav_accum
   @
   OCN -> MED :remapMethod=redist
   WAV -> MED :remapMethod=redist
   MED med_phases_post_ocn
   MED med_phases_post_wav
   MED med_phases_restart_write
@
::

But, dt_atmos in model_configure is 360s in current develop vs 720s before 2363. Was it intended for 2363 to reduce timestep for this test? Either way, now FV3<-->CMEPS no longer couple every ATM timestep, ATM timestep is different than ICE timestep,... I'm not sure this is intentional and may need hot fix if not intended.

@dpsarmie
Copy link
Collaborator Author

dpsarmie commented Sep 23, 2024

Ok I see what you mean.

#2363 was intended to change the timesteps to have the more optimized set of parameter values to allow for more stability at longer timesteps: NOAA-EMC/global-workflow#2574

I'm going to move forward with your idea of having DT_ATMOS set in export_fv3 and changed in export_ugwpv1 (if needed) before calling export_cpl.

That should allow DT_CICE and coupling_interval_fast_sec to be set correctly when the other export calls are run from within export_cpl.

Line 1302 will also be removed in default_vars.sh since DT_INNER will have been set by then.

Let me know if that logic makes sense. That should resolve the issues of not having the timing in sync between the models, correct?

@NickSzapiro-NOAA
Copy link
Collaborator

That makes sense to me. Maybe can ask Denise to review solution too

@dpsarmie
Copy link
Collaborator Author

dpsarmie commented Oct 8, 2024

Latest changes do the following:

To fix the timing issue it CMEPS, default_vars.sh has been reworked. If you export DT_ATMOS and/or ATMRES before the export_fv3 (and other export functions), then all dependent variables will be auto-populated. If these variables are not set, then defaults will be set. You can still manually set these parameters after the function calls but you need to make sure that all variables that depend on DT_ATMOS and ATMRES are also set unless you intended to do so otherwise.

Either way, the timing with CMEPS should be functioning as intended.

@dpsarmie dpsarmie marked this pull request as ready for review October 8, 2024 12:49
@jkbk2004
Copy link
Collaborator

@dpsarmie I am updating bl_date: 20241028.

@jkbk2004
Copy link
Collaborator

@dpsarmie I pushed hera log. But I noticed computational time increases for quite some cases. Can you review and confirm? @NickSzapiro-NOAA @junwang-noaa @BrianCurtis-NOAA FYI

junwang-noaa
junwang-noaa previously approved these changes Oct 29, 2024
@jkbk2004 jkbk2004 added the Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. label Oct 29, 2024
@dpsarmie
Copy link
Collaborator Author

Yes I see. I'm going to go through the tests and see if we can get them all back down (future PR). Most of the increases were from ugwpv1 and fixing the CICE/coupling timestep, I believe.

@jkbk2004
Copy link
Collaborator

OK! we can merge this pr. any final comment?

@jkbk2004 jkbk2004 merged commit 547be6d into ufs-community:develop Oct 31, 2024
4 checks passed
tsga added a commit to tsga/ufs-weather-model that referenced this pull request Nov 3, 2024
* develop:
  GW-RT global_control.nml modifications (ufs-community#2425)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
8 participants