Consolidate python configuration dictionaries #2839

DavidHuber-NOAA · 2024-08-16T14:59:45Z

Description

This modifies the way the config dictionary is constructed and referenced. Rather than updating a single configuration dictionary with each RUN, a RUN-based dictionary of config dictionaries is created and referenced by the appropriate RUN when calculating resources.

This also makes the methods that were hidden before #2727 hidden again.

Resolves #2783

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

XMLs for all CI tests were identical to develop on

Orion
Hercules
Hera
WCOSS2

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes

DavidHuber-NOAA · 2024-08-16T15:01:37Z

I still need to repeat the XML tests on WCOSS2, Hera, and Hercules, but this should be ready for review. When ready, I don't think this needs to go through CI or, at most, just through the setup stage.

* origin/develop: support ATM forecast only on Azure (NOAA-EMC#2827) Convert staging job to python and yaml (NOAA-EMC#2651) Fixed test on UNAVAILBLE in python Rocoto check (NOAA-EMC#2842)

DavidHuber-NOAA · 2024-08-21T17:24:54Z

I was able to repeat the XML tests on all platforms. All were identical. This PR is now ready to be reviewed.

DavidHuber-NOAA · 2024-08-21T17:26:19Z

@aerorahul I just recalled the conversation yesterday about the _ dictionary entry. I will update that presently and retest on Hera.

DavidHuber-NOAA · 2024-08-21T18:32:18Z

@aerorahul The no-RUN configs key has been renamed now.

WalterKolczynski-NOAA · 2024-08-22T06:45:05Z

workflow/applications/applications.py

-        # Save the configuration so we can source the config files when
-        # determining task resources
-        self.conf = conf
+        self._base = conf.parse_config('config.base')


Don't like making this an object field. This version should not be needed beyond initialization. All later uses should be using self.configs[run]['base'].

Suggested change

self._base = conf.parse_config('config.base')

base = conf.parse_config('config.base')

Alright, I removed the _base attribute from the AppConfig class and subclasses. _no_run is no longer referenced anywhere outside of _init_finalize. This is still required to determine which runs are valid for the experiment.

workflow/applications/applications.py

workflow/rocoto/tasks.py

* origin/develop: Fix gdas build on Gaea and add Gaea to available CI list (NOAA-EMC#2857) Support ATM forecast only on Google (NOAA-EMC#2832) Add GEFS C48 support on AWS (NOAA-EMC#2818) Update omega calculation (NOAA-EMC#2751) Add snow DA update and recentering for the EnKF forecasts (NOAA-EMC#2690)

* origin/develop: Add 3 and 9 hr increment files to IC staging (NOAA-EMC#2876) Add diffusion/diag B for aerosol DA and some other needed changes (NOAA-EMC#2738) Correct ocean `MOM.res_#` stage copy (NOAA-EMC#2868) Support coupling on AWS (NOAA-EMC#2859) Add JEDI ATM lgetkf observer and solver jobs (NOAA-EMC#2833)

WalterKolczynski-NOAA · 2024-09-04T02:30:35Z

Deferring any final elimination of the no_run to a future PR.

emcbot · 2024-09-04T02:35:16Z

CI Update on Wcoss2 at 09/04/24 02:35:15 AM
============================================
Cloning and Building global-workflow PR: 2839
with PID: 202173 on host: clogin03

emcbot · 2024-09-04T03:21:10Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Wed Sep  4 02:39:51 UTC 2024 on clogin03
---------------------------------------------------
Build: Completed at 09/04/24 03:20:46 AM
Case setup: Completed for experiment C48_ATM_92dfc754
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_92dfc754
Case setup: Skipped for experiment C48_S2SWA_gefs_92dfc754
Case setup: Completed for experiment C48_S2SW_92dfc754
Case setup: Completed for experiment C96_atm3DVar_extended_92dfc754
Case setup: Skipped for experiment C96_atm3DVar_92dfc754
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_92dfc754
Case setup: Completed for experiment C96C48_hybatmDA_92dfc754
Case setup: Completed for experiment C96C48_ufs_hybatmDA_92dfc754

emcbot · 2024-09-09T16:28:08Z

CI Update on Wcoss2 at 09/09/24 04:28:07 PM
============================================
Cloning and Building global-workflow PR: 2839
with PID: 204754 on host: dlogin03

emcbot · 2024-09-09T17:20:15Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Mon Sep  9 16:31:28 UTC 2024 on dlogin03
---------------------------------------------------
Build: Completed at 09/09/24 05:19:50 PM
Case setup: Completed for experiment C48_ATM_1efa7c59
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_1efa7c59
Case setup: Skipped for experiment C48_S2SWA_gefs_1efa7c59
Case setup: Completed for experiment C48_S2SW_1efa7c59
Case setup: Completed for experiment C96_atm3DVar_extended_1efa7c59
Case setup: Skipped for experiment C96_atm3DVar_1efa7c59
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_1efa7c59
Case setup: Completed for experiment C96C48_hybatmDA_1efa7c59
Case setup: Completed for experiment C96C48_ufs_hybatmDA_1efa7c59

emcbot · 2024-09-09T21:14:43Z

CI Passed on Hera in Build# 4
Built and ran in directory /scratch1/NCEPDEV/global/CI/2839


Experiment C48_ATM_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 19:02:46 UTC 2024
Experiment C48mx500_3DVarAOWCDA_1efa7c59 Completed 2 Cycles: *SUCCESS* at Mon Sep  9 19:02:49 UTC 2024
Experiment C96C48_hybatmDA_1efa7c59 Completed 3 Cycles: *SUCCESS* at Mon Sep  9 20:18:12 UTC 2024
Experiment C96_atm3DVar_1efa7c59 Completed 3 Cycles: *SUCCESS* at Mon Sep  9 20:23:02 UTC 2024
Experiment C96C48_ufs_hybatmDA_1efa7c59 Completed 2 Cycles: *SUCCESS* at Mon Sep  9 20:36:33 UTC 2024
Experiment C48_S2SWA_gefs_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 20:36:58 UTC 2024
Experiment C48_S2SW_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 20:54:17 UTC 2024
Experiment C96C48_hybatmaerosnowDA_1efa7c59 Completed 3 Cycles: *SUCCESS* at Mon Sep  9 21:14:35 UTC 2024

emcbot · 2024-09-09T21:31:12Z

CI Passed on Orion in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/ORION/2839


Experiment C48_ATM_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 02:46:50 PM CDT 2024
Experiment C96C48_hybatmDA_1efa7c59 Completed 3 Cycles: *SUCCESS* at Mon Sep  9 04:06:37 PM CDT 2024
Experiment C96_atm3DVar_1efa7c59 Completed 3 Cycles: *SUCCESS* at Mon Sep  9 04:12:45 PM CDT 2024
Experiment C48_S2SWA_gefs_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 04:20:16 PM CDT 2024
Experiment C48_S2SW_1efa7c59 Completed 1 Cycles: *SUCCESS* at Mon Sep  9 04:31:03 PM CDT 2024

emcbot · 2024-09-10T06:30:08Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_1efa7c59 *** SUCCESS *** at 09/09/24 06:57:12 PM
Experiment C48_S2SW_1efa7c59 *** SUCCESS *** at 09/09/24 07:12:12 PM
Experiment C96C48_hybatmDA_1efa7c59 *** SUCCESS *** at 09/09/24 07:57:23 PM
Experiment C96C48_hybatmaerosnowDA_1efa7c59 *** SUCCESS *** at 09/09/24 08:51:25 PM
Experiment C96C48_ufs_hybatmDA_1efa7c59 *** SUCCESS *** at 09/09/24 09:51:19 PM
Experiment C96_atm3DVar_extended_1efa7c59 *** SUCCESS *** at 09/10/24 06:27:35 AM

emcbot · 2024-09-18T14:00:30Z

Experiment C96_atm3DVar_extended_92dfc754 UNKNOWN on Wcoss2 at 09/18/24 02:00:29 PM

Error logs:

Follow link here to view the contents of the above file(s): (link)

DavidHuber-NOAA added 2 commits August 16, 2024 09:39

Construct dict of dicts of config files NOAA-EMC#2783

08f558e

Add comment for sourcing configs based on RUN

0e60123

DavidHuber-NOAA requested review from aerorahul and WalterKolczynski-NOAA August 16, 2024 14:59

Merge remote-tracking branch 'origin/develop' into fix/early_run_configs

59c2002

* origin/develop: support ATM forecast only on Azure (NOAA-EMC#2827) Convert staging job to python and yaml (NOAA-EMC#2651) Fixed test on UNAVAILBLE in python Rocoto check (NOAA-EMC#2842)

DavidHuber-NOAA added 2 commits August 21, 2024 18:16

Rename no-RUN configs key

e06d807

Clean up applications.py

15c2883

WalterKolczynski-NOAA requested changes Aug 22, 2024

View reviewed changes

DavidHuber-NOAA added 4 commits August 28, 2024 15:08

Avoid adding _base to app objects

52f448c

Declare 'run' attribute in GEFSAppConfig

92dfc75

WalterKolczynski-NOAA previously approved these changes Sep 4, 2024

View reviewed changes

WalterKolczynski-NOAA added the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Sep 4, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 4, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Sep 4, 2024

WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Sep 4, 2024

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Sep 4, 2024

WalterKolczynski-NOAA added the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Sep 9, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Sep 9, 2024

emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Sep 9, 2024

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Sep 10, 2024

Merge branch 'develop' into fix/early_run_configs

514a9fd

WalterKolczynski-NOAA approved these changes Sep 10, 2024

View reviewed changes

WalterKolczynski-NOAA merged commit b443915 into NOAA-EMC:develop Sep 10, 2024
3 of 4 checks passed

emcbot added the CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed label Sep 18, 2024

DavidHuber-NOAA deleted the fix/early_run_configs branch November 4, 2024 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate python configuration dictionaries #2839

Consolidate python configuration dictionaries #2839

DavidHuber-NOAA commented Aug 16, 2024 •

edited

Loading

DavidHuber-NOAA commented Aug 16, 2024

DavidHuber-NOAA commented Aug 21, 2024

DavidHuber-NOAA commented Aug 21, 2024

DavidHuber-NOAA commented Aug 21, 2024

WalterKolczynski-NOAA Aug 22, 2024

DavidHuber-NOAA Aug 30, 2024

WalterKolczynski-NOAA commented Sep 4, 2024

emcbot commented Sep 4, 2024

emcbot commented Sep 4, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 10, 2024

emcbot commented Sep 18, 2024

	self._base = conf.parse_config('config.base')
	base = conf.parse_config('config.base')

Consolidate python configuration dictionaries #2839

Consolidate python configuration dictionaries #2839

Conversation

DavidHuber-NOAA commented Aug 16, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidHuber-NOAA commented Aug 16, 2024

DavidHuber-NOAA commented Aug 21, 2024

DavidHuber-NOAA commented Aug 21, 2024

DavidHuber-NOAA commented Aug 21, 2024

WalterKolczynski-NOAA Aug 22, 2024

Choose a reason for hiding this comment

DavidHuber-NOAA Aug 30, 2024

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Sep 4, 2024

emcbot commented Sep 4, 2024

emcbot commented Sep 4, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 9, 2024

emcbot commented Sep 10, 2024

emcbot commented Sep 18, 2024

DavidHuber-NOAA commented Aug 16, 2024 •

edited

Loading