Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Netcdf output #54

Merged
merged 5 commits into from
May 9, 2023
Merged

Netcdf output #54

merged 5 commits into from
May 9, 2023

Conversation

sjdaines
Copy link
Member

@sjdaines sjdaines commented May 9, 2023

Add netCDF output using PALEOmodel.OutputWriters.save_netcdf and PALEOmodel.OutputWriters.load_netcdf!.

This solves several related problems:

  • future-proof PALEOmodel output (output written now can be read by future PALEO versions)
  • interoperability eg with Python via xarray
  • allow internal data formats to be changed without breaking compatibility with old saved output, ideally to use a standard Julia package (see Use a generic xarray-like Julia package for output #53)

This is now the recommended way to save/load model output. Use of Julia-specific jld2 output with PALEOmodel.OutputWriters.save_jld2 and PALEOmodel.OutputWriters.load_jld2 now emits a deprecation and future-compatibility warning.

Currently can write variables from scalar Domains only (this is not checked!)

TODO
 - type conversion for attributes
 - read scalar domains
 - scalar variables in non-scalar domains
 - grids
Write out
    UnstructuredVectorGrid
    UnstructuredColumnGrid

    CellSpace variables at least for these two grids (one dimension 'cells')
Can now save/load
  PB.Grids.UnstructuredVectorGrid
  PB.Grids.UnstructuredColumnGrid

TODO
 - cartesian grids
 - add test cases to PALEOexamples
 - update doc

 Known issues / limitations:
 - PALEO currently has no concept of constant (time independent)
   variables. As a workaround, test in `variable_is_constant`
   looks for variable attribute :datatype = Float64 (and checks data is constant),
   where this attribute usually means constant but can also be used to
   label variables that won't be included in an AD Jacobian

- NamedDimension (used in UnstructuredColumnGrid and for Domain
  data_dims eg used for a wavelength grid) also includes FixedCoordinate
  variables, which may not be present in the output and may not even be
  uniquely names. As a workaround, for data_dims generate unique names and save/load an
  extra variable, for UnstructuredColumnGrid assume variable will be present
  (TODO this should be changed for consistency).

- Coordinate variables are not labelled with CF etc attributes

- UnstructuredColumnGrid uses a non-standard way of saving indices
  (similar to contiguous ragged array in the CF spec, but unlikely to
  be supported by other tools)
- can now save/load most grid types, except CartesianArrayGrid
- tests added to PALEOdev.jl
- Jupyter notebooks with Python xarray examples added to PALEOdev.jl
@sjdaines sjdaines merged commit f76228c into PALEOtoolkit:main May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant