marine post & vrfy bug fixes

[JohnSteffen-NOAA]

This issue outlines several known bugs and potential fixes related to the marine post & vrfy tasks.

1. The ocean grid input file "ocngrid.nc" is missing and not properly copied/staged in the marine post task.

From the gdasocnanalvrfy.log:

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210701/00/analysis/ocean/gdas.t21z.ocngrid.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'a7cdcd55-3299-4e6b-b1ee-07ec6b18a1a6']


FileNotFoundError: [Errno 2] No such file or directory: 
'/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210701/00/analysis/ocean/gdas.t21z.ocngrid.nc'

Solution:
Modify the /scripts/exgdas_global_marine_analysis_post.py script to uncomment the line that copies the soca_gridspec.nc file from the run directory to the COMROOT directory.

 post_file_list.append([os.path.join(anl_dir, 'soca_gridspec.nc'),
                        os.path.join(com_ocean_analysis, f'{RUN}.t{bcyc}z.ocngrid.nc')])

2. Two variables are not defined in the marine_eva_post.py script: waterTemperature and salinity. See this previously reported issue here and the proposed fix: #1153

3. Ice variable names are inconsistent between the netcdf files and the exgdas_global_marine_analysis_vrfy.py script (ie aicen vs. aice_h)

From the gdasocnanalvrfy.log:

"/scratch1/NCEPDEV/da/John.Steffen/global-workflow-082924-update/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.py", line 49, in plot_marine_vrfy
    ocnvrfyPlotter.plot()
  File "/scratch1/NCEPDEV/da/John.Steffen/global-workflow-082924-update/sorc/gdas.cd/ush/soca/soca_vrfy.py", line 292, in plot
    plotHorizontalSlice(self.config)
  File "/scratch1/NCEPDEV/da/John.Steffen/global-workflow-082924-update/sorc/gdas.cd/ush/soca/soca_vrfy.py", line 92, in plotHorizontalSlice
    slice_data = np.squeeze(data[variable])
  File "/scratch1/NCEPDEV/da/python/envs/eva/lib/python3.10/site-packages/xarray/core/dataset.py", line 1547, in __getitem__
    raise KeyError(
KeyError: "No variable named 'aicen'. Variables on the dataset include ['xaxis_1', 'yaxis_1', 'Time', 'aice_h', 'hi_h', 'hs_h']"

Solution:
Modify the exgdas_global_marine_analysis_vrfy.py script to use correct/updated variable names (aice_h, hi_h, hs_h).

            plotConfig(grid_file=grid_file,
                       data_file=os.path.join(comout, f'{RUN}.t'+cyc+'z.ice.incr.nc'),
                       lats=np.arange(-60, 60, 10),
-                      variables_horiz={'aicen': [-0.2, 0.2],
-                                       'hicen': [-0.5, 0.5],
-                                       'hsnon': [-0.1, 0.1]},
+                      variables_horiz={'aice_h': [-0.2, 0.2],
+                                       'hi_h': [-0.5, 0.5],
+                                       'hs_h': [-0.1, 0.1]},


            plotConfig(grid_file=grid_file,
                       data_file=os.path.join(comout, f'{RUN}.t'+cyc+'z.iceana.nc'),
-                      variables_horiz={'aicen': [0.0, 1.0],
-                                       'hicen': [0.0, 4.0],
-                                       'hsnon': [0.0, 0.5]},
+                      variables_horiz={'aice_h': [0.0, 1.0],
+                                       'hi_h': [0.0, 4.0],
+                                       'hs_h': [0.0, 0.5]},

4. The *bkgerr_stddev.nc file is not copied from the run directory to COMROOT.

From gdasocnanalpost.log:

OSError: unable to copy /scratch1/NCEPDEV/stmp2/John.Steffen/RUNDIRS/marine_candidate_092024/gdas.2021071400/gdasocnanal_00/ocean.bkgerr_stddev.inc.nc to /scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210714/00//analysis/ocean/gdas.t00z.ocn.bkgerr_stddev.nc

Solution:
Uncomment and modify the following lines in the exgdas_global_marine_analysis_post.py script.

post_file_list.append([os.path.join(anl_dir, f'ocean.bkgerr_stddev.nc'),
                           os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.bkgerr_stddev.nc')])

5. Some of the other stats *.nc files are missing

From gdasocnanalvrfy.log:

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210718/00/analysis/ocean/gdas.t00z.ocn.ssh_steric_stddev.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '4064cbe5-094a-4545-90a7-05261ef2648b']
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210718/00/analysis/ocean/gdas.t00z.ocn.ssh_total_stddev.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'f3e07b76-bc45-4f50-b726-93ec119ff62b']
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210718/00/analysis/ocean/gdas.t00z.ocn.recentering_error.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'b15dcb30-3e5d-4b7e-9305-d88764ca07b3']


FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210718/00/analysis/ocean/gdas.t00z.ocn.ssh_unbal_stddev.nc'
FileNotFoundError: [Errno 2] No such file or directory: '/scratch1/NCEPDEV/da/John.Steffen/expts/marine_candidate_092024/marine_candidate_092024/COMROOT/marine_candidate_092024/gdas.20210718/00/analysis/ocean/gdas.t00z.ocn.steric_explained_variance.nc'

@apchoiCMD needs to find the solution

6. Marine vrfy task is hanging and reaching the wall clock limit.

From gdasocnanalvrfy.log:

  warnings.warn(f'Downloading: {url}', DownloadWarning)
slurmstepd: error: *** JOB 482666 ON h34m28 CANCELLED AT 2024-10-03T18:35:59 DUE TO TIME LIMIT ***

Possible solutions:
Job is hanging during downloading the cartopy raster and shapefiles for coastlines. Instead, copy these files to the common folder and point to them inside the soca_vrfy.py script.

7. The gdassoca_obsstats is failing during the marine post task, likely related to the in situ obs statistics

From the gdasocnanalpost.log:

2024-10-03 10:57:21,534 - WARNING  - root        : srun -l --export=ALL -n 1 /scratch1/NCEPDEV/da/John.Steffen/global-workflow-082924-update/sorc/gdas.cd/build/bin/gdassoca_obsstats.x diag_stats.yaml has failed exit 1

Solution:
The gdassoca_obsstats executable needs the ocean basin name to be defined

[guillaumevernieres]

The issue is a missing raster file that cartopy is trying to download from a compute node.