Update the IMS pre-processing time and IMS snow data format #2779

jiaruidong2017 · 2024-07-19T13:16:27Z

Description

Since the IMS snow products was ingested at about 23:00 UTC, and the IMS snow cover data is in the ascii format. Therefore, we propose to assimilate the IMS snow cover data in the ascii format at 00z cycle.

This PR updates the IMS snow data pre-processing time from being the 18z to the 00z cycle, and also changes to assimilate the IMS snow data in ascii format from the operational dump.

An IMS snow data in ascii format will be staged to global dump directory for the CI test C96_atmaerosnowDA.yaml at:
/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump/gdas.20211221/00/atmos/gdas.t00z.imssnow96.asc

This PR depends on NOAA-EMC/GDASApp#1228

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

aerorahul

Conditionally approve pending CI

This PR updates the IMS snow data pre-processing time from being the 18z to the 00z cycle, and also changes to assimilate the IMS snow data in ascii format from the operational dump. This PR contributes to NOAA-EMC/global-workflow#2779

CoryMartin-NOAA · 2024-07-19T13:33:13Z

scripts/exglobal_prep_snow_obs.py

@@ -21,5 +21,6 @@
    # Instantiate the snow prepare task
    SnowAnl = SnowAnalysis(config)
    SnowAnl.prepare_GTS()
-    if f"{ SnowAnl.task_config.cyc }" == '18':
+    SnowAnl.task_config.cyc = f'{SnowAnl.task_config.cyc:02d}'


Why is this needed?

To keep the leading '0' for the 00z cycle to be '00', so that the following if f"{ SnowAnl.task_config.cyc }" == '00': is true for the 00z

can't it just be if SnowAnl.task_config.cyc == 0?

Made the according changes. Thanks.

CoryMartin-NOAA · 2024-07-19T14:29:23Z

Looks good, the g-w team will need to work with @jiaruidong2017 to update the test data before kicking off CI

aerorahul · 2024-07-19T14:33:30Z

@KateFriedman-NOAA @WalterKolczynski-NOAA
Test data needs to be updated as part of this PR.
Please coordinate w/ @jiaruidong2017

jiaruidong2017 · 2024-07-19T14:37:11Z

@KateFriedman-NOAA @WalterKolczynski-NOAA Would you please copy the test data:
/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump/gdas.20211221/00/atmos/gdas.t00z.imssnow96.asc on hera to the global dump for CI test. Thanks.

KateFriedman-NOAA · 2024-07-22T13:06:30Z

@jiaruidong2017 I have copied the file into the GDA on Hera:

[role.glopara@hfe11 atmos]$ pwd
/scratch1/NCEPDEV/global/glopara/dump/gdas.20211221/00/atmos
[role.glopara@hfe11 atmos]$ ll *ims*
-rw-r--r-- 1 role.glopara global 37756368 Jul 17 19:22 gdas.t00z.imssnow96.asc
-rw-r--r-- 1 role.glopara global 10259780 Dec 21  2021 gdas.t00z.imssnow96.grib2

I have also synced this to the GDA on both WCOSS2s. @WalterKolczynski-NOAA please sync this file to the GDA on Orion, thanks!

RussTreadon-NOAA · 2024-07-22T18:03:18Z

We also need to copy
/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump/gfs.20211221/00/atmos/gfs.t00z.imssnow96.asc

to

/scratch1/NCEPDEV/global/glopara/dump/gfs.20211221/00/atmos

The 20211221 00Z gfsprepsnowobs from g-w C96_atmaerosnowDA CI fails because gfs.t00z.imssnow96.asc can not be found.

jiaruidong2017 · 2024-07-22T18:20:40Z

We also need to copy /scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump/gfs.20211221/00/atmos/gfs.t00z.imssnow96.asc

to

/scratch1/NCEPDEV/global/glopara/dump/gfs.20211221/00/atmos

The 20211221 00Z gfsprepsnowobs from g-w C96_atmaerosnowDA CI fails because gfs.t00z.imssnow96.asc can not be found.

Thanks @RussTreadon-NOAA. Yes, @KateFriedman-NOAA would you please make the copy. Thanks.

CoryMartin-NOAA · 2024-07-22T21:09:42Z

I think this PR is needed to be merged to fix failures seen in CI testing on #2749

guillaumevernieres · 2024-07-22T22:22:45Z

Could you resolve the conflict @jiaruidong2017 ?

jiaruidong2017 · 2024-07-23T00:38:41Z

Could you resolve the conflict @jiaruidong2017 ?

@guillaumevernieres I learn to try it by merging branch develop. Is this correct way to do it?

KateFriedman-NOAA · 2024-07-23T13:08:48Z

@jiaruidong2017 @RussTreadon-NOAA I have also copied the gfs file into the GDA on Hera:

[role.glopara@hfe11 atmos]$ pwd
/scratch1/NCEPDEV/global/glopara/dump/gfs.20211221/00/atmos
[role.glopara@hfe11 atmos]$ ll *ims*
-rw-r--r-- 1 role.glopara global 37756368 Jul 17 23:15 gfs.t00z.imssnow96.asc
-rw-r--r-- 1 role.glopara global 10259780 Dec 21  2021 gfs.t00z.imssnow96.grib2

...and synced it to both WCOSS2s. @WalterKolczynski-NOAA please also sync this file to Orion, thanks!

RussTreadon-NOAA · 2024-07-23T13:43:20Z

Thank you @KateFriedman-NOAA for updating the GDA. I reran the failed 20211221 00Z gfsprepsnowobs from C96_atmaerosnowDA CI. The job successfully ran to completion.

WalterKolczynski-NOAA · 2024-07-23T14:15:07Z

Synchronized to MSU.

emcbot · 2024-07-23T14:20:14Z

CI Update on Wcoss2 at 07/23/24 02:20:14 PM
============================================
Cloning and Building global-workflow PR: 2779
with PID: 193286 on host: clogin03

emcbot · 2024-07-23T14:21:48Z

Failed on cloning and building global-workflowi PR: 2779
CI on Wcoss2 failed to build on Tue Jul 23 14:21:45 UTC 2024 for repo [email protected]:NOAA-EMC/global-workflow.git

emcbot · 2024-07-23T14:42:52Z

Build FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2779/gfs/sorc/logs/build_gdas.log

Follow link here to view the contents of the above file(s): (link)

CoryMartin-NOAA · 2024-07-23T14:58:31Z

@guillaumevernieres any idea what happened here? Did something change in SOCA with MOM6?

RussTreadon-NOAA · 2024-07-23T15:45:10Z

@jiaruidong2017 , did you intend to update the sorc/gdas.cd hash when you merged g-w develop into jiaruidong2017:feature/ims_ascii at d10c3f5?

jiaruidong2017 · 2024-07-23T16:07:58Z

@RussTreadon-NOAA

@guillaumevernieres is working on this to merge my changes to his branch.

RussTreadon-NOAA · 2024-07-23T16:10:42Z

Thank you @jiaruidong2017 for the update.

Thank you @guillaumevernieres for merging PR #2779 into #2749.

DavidHuber-NOAA · 2024-07-23T16:43:57Z

Manually killed CI tests on Hera and Hercules which had stalled. Will restart testing on #2749 when merged and approved.

jiaruidong2017 · 2024-07-23T18:41:52Z

This PR will be merged into the PR #2749.

jiaruidong2017 added 2 commits July 17, 2024 15:50

Change to process IMS snow from the 18z to the 00z cycle.

ba3338f

Include the changes from GDASApp.

15fb45e

jiaruidong2017 mentioned this pull request Jul 19, 2024

Update the IMS pre-processing time and IMS snow data format NOAA-EMC/GDASApp#1228

Merged

aerorahul previously approved these changes Jul 19, 2024

View reviewed changes

aerorahul requested a review from CoryMartin-NOAA July 19, 2024 13:28

CoryMartin-NOAA reviewed Jul 19, 2024

View reviewed changes

Address reviewer's comments.

7773f81

jiaruidong2017 dismissed aerorahul’s stale review via 7773f81 July 19, 2024 13:58

Update the changes from GDASApp.

152e3a8

CoryMartin-NOAA previously approved these changes Jul 19, 2024

View reviewed changes

CoryMartin-NOAA added the Fix Files label Jul 19, 2024

Merge branch 'develop' into feature/ims_ascii

d10c3f5

jiaruidong2017 dismissed CoryMartin-NOAA’s stale review via d10c3f5 July 23, 2024 00:35

jiaruidong2017 mentioned this pull request Jul 23, 2024

Refactoring of the marine B-matrix job #2749

Merged

17 tasks

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jul 23, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jul 23, 2024

DavidHuber-NOAA removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Jul 23, 2024

jiaruidong2017 closed this Jul 23, 2024

jiaruidong2017 deleted the feature/ims_ascii branch July 25, 2024 00:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the IMS pre-processing time and IMS snow data format #2779

Update the IMS pre-processing time and IMS snow data format #2779

jiaruidong2017 commented Jul 19, 2024 •

edited

Loading

aerorahul left a comment

CoryMartin-NOAA Jul 19, 2024

jiaruidong2017 Jul 19, 2024

CoryMartin-NOAA Jul 19, 2024

jiaruidong2017 Jul 19, 2024

CoryMartin-NOAA commented Jul 19, 2024

aerorahul commented Jul 19, 2024

jiaruidong2017 commented Jul 19, 2024

KateFriedman-NOAA commented Jul 22, 2024

RussTreadon-NOAA commented Jul 22, 2024

jiaruidong2017 commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

guillaumevernieres commented Jul 22, 2024

jiaruidong2017 commented Jul 23, 2024

KateFriedman-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

WalterKolczynski-NOAA commented Jul 23, 2024

emcbot commented Jul 23, 2024

emcbot commented Jul 23, 2024

emcbot commented Jul 23, 2024

CoryMartin-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

jiaruidong2017 commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

DavidHuber-NOAA commented Jul 23, 2024

jiaruidong2017 commented Jul 23, 2024

Update the IMS pre-processing time and IMS snow data format #2779

Update the IMS pre-processing time and IMS snow data format #2779

Conversation

jiaruidong2017 commented Jul 19, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

aerorahul left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA Jul 19, 2024

Choose a reason for hiding this comment

jiaruidong2017 Jul 19, 2024

Choose a reason for hiding this comment

CoryMartin-NOAA Jul 19, 2024

Choose a reason for hiding this comment

jiaruidong2017 Jul 19, 2024

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jul 19, 2024

aerorahul commented Jul 19, 2024

jiaruidong2017 commented Jul 19, 2024

KateFriedman-NOAA commented Jul 22, 2024

RussTreadon-NOAA commented Jul 22, 2024

jiaruidong2017 commented Jul 22, 2024

CoryMartin-NOAA commented Jul 22, 2024

guillaumevernieres commented Jul 22, 2024

jiaruidong2017 commented Jul 23, 2024

KateFriedman-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

WalterKolczynski-NOAA commented Jul 23, 2024

emcbot commented Jul 23, 2024

emcbot commented Jul 23, 2024

emcbot commented Jul 23, 2024

CoryMartin-NOAA commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

jiaruidong2017 commented Jul 23, 2024

RussTreadon-NOAA commented Jul 23, 2024

DavidHuber-NOAA commented Jul 23, 2024

jiaruidong2017 commented Jul 23, 2024

jiaruidong2017 commented Jul 19, 2024 •

edited

Loading