Move the variational scripts to this repository #2920

guillaumevernieres · 2024-09-13T18:12:37Z

Description

This PR "moves and refactors" the variational DA exscripts that were in the GDASapp to this repository. The ens. var. feature will be replicated/moved in a subsequent PR.

fixes #Refactoring of the marine DA scripts GDASApp#977

Waiting for 2 companion PR's:

ci testing: We should be able to turn on the WCDA test on Hercules

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

The latest tests were done on Hecules, earlier ones on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ci/cases/gfsv17/ocnanal.yaml

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT

ush/python/pygfs/task/marine_bmat.py

ush/python/pygfs/utils/marine_da_utils.py

DavidNew-NOAA

Looks good to me. No comments beyond what others have already said.

ush/python/pygfs/utils/marine_da_utils.py

ush/python/pygfs/task/marine_analysis.py

This PR needs to be merged before - [NOAA-EMC/global-workflow#2920 It depends on - [NOAA-EMC/jcb-gdas#26 and will require an update to the `jcb-gdas` submodules. It will break the GDASApp marine CI until the `g-w` pr is merged.

emcbot · 2024-09-19T21:50:51Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_2a5e9bff/logs/2021032418/gdasmarineanlinit.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-09-19T21:50:57Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_2a5e9bff

emcbot · 2024-09-19T21:57:06Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_2a5e9bff/logs/2021122018/gdasprepsnowobs.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-09-19T21:57:11Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_2a5e9bff

RussTreadon-NOAA · 2024-09-19T23:14:22Z

C48mx500_3DVarAOWCDA job gdasmarineanlinit.log failed with

  File "/scratch1/NCEPDEV/global/CI/2920/gfs/ush/python/pygfs/utils/marine_da_utils.py", line 86, in test_hist_date
    ncf = Dataset(histfile, 'r')
NameError: name 'Dataset' is not defined
+ JGLOBAL_MARINE_ANALYSIS_INITIALIZE[40]: status=1
+ JGLOBAL_MARINE_ANALYSIS_INITIALIZE[41]: [[ 1 -ne 0 ]]
+ JGLOBAL_MARINE_ANALYSIS_INITIALIZE[41]: exit 1

RussTreadon-NOAA · 2024-09-19T23:17:09Z

C96C48_hybatmaerosnowDA may be fixed if we update guillaumevernieres:feature/marinevar with the current head of g-w develop. guillaumevernieres:feature/marinevar is one commit behind develop

emcbot · 2024-09-20T01:07:21Z

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/2920


Experiment C48mx500_3DVarAOWCDA_2a5e9bff Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Sep 19 21:50:32 UTC 2024
Experiment C48mx500_3DVarAOWCDA_2a5e9bff Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_2a5e9bff/logs/2021032418/gdasmarineanlinit.log
Experiment C96C48_hybatmaerosnowDA_2a5e9bff Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Sep 19 21:56:51 UTC 2024
Experiment C96C48_hybatmaerosnowDA_2a5e9bff Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/2920/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_2a5e9bff/logs/2021122018/gdasprepsnowobs.log
Experiment C48_ATM_2a5e9bff Completed 1 Cycles: *SUCCESS* at Thu Sep 19 23:15:57 UTC 2024
Experiment C96_atm3DVar_2a5e9bff Completed 3 Cycles: *SUCCESS* at Fri Sep 20 00:17:24 UTC 2024
Experiment C96C48_hybatmDA_2a5e9bff Completed 3 Cycles: *SUCCESS* at Fri Sep 20 00:17:27 UTC 2024
Experiment C48_S2SWA_gefs_2a5e9bff Completed 1 Cycles: *SUCCESS* at Fri Sep 20 00:42:25 UTC 2024
Experiment C48_S2SW_2a5e9bff Completed 1 Cycles: *SUCCESS* at Fri Sep 20 00:47:52 UTC 2024
Experiment C96C48_ufs_hybatmDA_2a5e9bff Completed 2 Cycles: *SUCCESS* at Fri Sep 20 01:07:13 UTC 2024

guillaumevernieres · 2024-09-20T10:42:44Z

Merged develop and pushed a fix for the marine_utils, any chance you could trigger the ci again @aerorahul ?

aerorahul · 2024-09-20T17:26:46Z

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

+# Generate COM variables from templates
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_OBS
+
+RUN=${GDUMP} YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \


Why is this GDUMP? This should be RUN.

GDUMP is associated with the guess; that is the previous cycle gPDY/gcyc -- just clarifying why the q.

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

aerorahul · 2024-09-20T17:32:20Z

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

-export CDUMP=${CDUMP:-${RUN:-"gfs"}}
-export CDATE=${CDATE:-${PDY}${cyc}}
+# Ignore possible spelling error (nothing is misspelled)
+# shellcheck disable=SC2153
 export GDUMP=${GDUMP:-"gdas"}


I doubt, GDUMP is used in a finalize step, but it might be needed in the __init__. Just check if that is the case.

aerorahul · 2024-09-20T17:45:12Z

jobs/rocoto/marineanlchkpt.sh

@@ -8,11 +8,10 @@ source "${HOMEgfs}/ush/preamble.sh"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"

-export job="ocnanalrun"
-export jobid="${job}.$$"
+export job="marineanlchkpt"


why remove jobid?

no reason, will fix.

Co-authored-by: Rahul Mahajan <[email protected]>

guillaumevernieres and others added 7 commits August 28, 2024 09:11

...

c9e41b1

var init ok, run needs tlc

53a4de8

updated checkpoint task

ba78b00

working finalize

4509b0c

Merge branch 'develop' into feature/marinevar

2b667f2

...

317e351

removed envar test yaml

ea6ae91

guillaumevernieres requested review from aerorahul, WalterKolczynski-NOAA and AndrewEichmann-NOAA September 13, 2024 18:12

github-advanced-security bot found potential problems Sep 13, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

jobs/JGLOBAL_MARINE_ANALYSIS_CHECKPOINT Fixed Show fixed Hide fixed

guillaumevernieres added 4 commits September 13, 2024 13:30

norms, try 1

5c7eb42

Merge branch 'develop' into feature/marinevar

b4d46aa

norms, try 2

3b15dc6

norms, try 3

90ad1d3

guillaumevernieres mentioned this pull request Sep 13, 2024

Move the variational scripts to the g-w NOAA-EMC/GDASApp#1282

Merged

AndrewEichmann-NOAA reviewed Sep 13, 2024

View reviewed changes

aerorahul requested a review from DavidNew-NOAA September 14, 2024 14:44

guillaumevernieres added 2 commits September 16, 2024 17:54

addressed review comments

f1d7f8b

Merge branch 'develop' into feature/marinevar

f84ae82

DavidNew-NOAA approved these changes Sep 17, 2024

View reviewed changes

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/utils/marine_da_utils.py Outdated Show resolved Hide resolved

aerorahul reviewed Sep 17, 2024

View reviewed changes

ush/python/pygfs/task/marine_analysis.py Show resolved Hide resolved

DavidNew-NOAA mentioned this pull request Sep 17, 2024

Incorporate JEDI class into marine jobs #2931

Open

guillaumevernieres mentioned this pull request Sep 18, 2024

Use jinja template for the obs list NOAA-EMC/GDASApp#1285

Open

guillaumevernieres added 2 commits September 18, 2024 11:00

module tidy

9e8bbad

Merge branch 'develop' into feature/marinevar

a286b1e

guillaumevernieres requested a review from aerorahul September 18, 2024 16:07

updated gdas and jcb-gdas hashes

a4d8391

emcbot removed the CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera label Sep 19, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Sep 19, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Sep 20, 2024

missing module

a2eb39e

aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Sep 20, 2024

Merge branch 'develop' into feature/marinevar

259df53

aerorahul added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Sep 20, 2024

aerorahul reviewed Sep 20, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE Outdated Show resolved Hide resolved

aerorahul reviewed Sep 20, 2024

View reviewed changes

jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE Outdated Show resolved Hide resolved

aerorahul reviewed Sep 20, 2024

View reviewed changes

guillaumevernieres and others added 2 commits September 20, 2024 16:49

Update jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

77f3ddc

Co-authored-by: Rahul Mahajan <[email protected]>

Update jobs/JGLOBAL_MARINE_ANALYSIS_FINALIZE

98ace06

Co-authored-by: Rahul Mahajan <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the variational scripts to this repository #2920

Move the variational scripts to this repository #2920

guillaumevernieres commented Sep 13, 2024 •

edited

Loading

DavidNew-NOAA left a comment

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

RussTreadon-NOAA commented Sep 19, 2024

RussTreadon-NOAA commented Sep 19, 2024

emcbot commented Sep 20, 2024

guillaumevernieres commented Sep 20, 2024

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

aerorahul Sep 20, 2024

guillaumevernieres Sep 20, 2024

Move the variational scripts to this repository #2920

Are you sure you want to change the base?

Move the variational scripts to this repository #2920

Conversation

guillaumevernieres commented Sep 13, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

emcbot commented Sep 19, 2024

RussTreadon-NOAA commented Sep 19, 2024

RussTreadon-NOAA commented Sep 19, 2024

emcbot commented Sep 20, 2024

guillaumevernieres commented Sep 20, 2024

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

aerorahul Sep 20, 2024

Choose a reason for hiding this comment

guillaumevernieres Sep 20, 2024

Choose a reason for hiding this comment

guillaumevernieres commented Sep 13, 2024 •

edited

Loading