netCDF-schema CF 1.11 implementation

[gtesoro]

Concept

The netCDF-Schema is new tool development aimed to provide an easy way to define and validate netCDF products.
The idea behind it is to have schemas, defined as yaml files, that describe the structure and relationships of a netCDF file. This schema follows the structure of a validation model, which will be used by the tool for validation.

The goal for the initial implementation is to have a validation model with all the necessary features so the CF Conventions (version 1.11 at the moment of writing) can be translated into an schema.

Here is the latest validation model and CF-11 schema.

How it works

Schemas have a root where all groups under validation are defined. Each of these groups may contain entries for dimensions, attributes and variables. Group name can be literals or regex expression to match more complex paths.

groups:

  # Root group
  - name: /
    attributes:

      # 2.6.1 Files that conform to the CF version 1.11 conventions must indicate this by setting the global Conventions attribute to contain the CF string value "CF-1.11"
      - criteria:
          name: Conventions
          value: "CF-1.11"
        count: 1
        comments: '2.6.1 Files that conform to the CF version 1.11 conventions must indicate this by setting the global Conventions attribute to contain the CF string value "CF-1.11"'

  # Any group except root
  - name: "[^/]+"
    attributes:
      # 2.7 The Conventions and external_variables attributes must not be used in non-root groups.
      - criteria:
          name: Conventions
        count: 0
        comments: '2.7 The Conventions and external_variables attributes must not be used in non-root groups.'

      - criteria:
          name: external_variables 
        count: 0
        comments: '2.7 The Conventions and external_variables attributes must not be used in non-root groups.'
  
  # Any group
  - name: '.+'
    attributes:

      - schema:
          name: 
            re: ^[A-Za-z][A-Za-z0-9_]*$ 
        comments: '2.3 Variable, dimension and attribute names should begin with a letter and be composed of letters (A-Z, a-z), digits (0-9), and underscores(_).'

      # 2.6.2 The title, history, institution, source, references, and comment attributes are all type string
      - criteria:
          name: title
        schema:
          type: str
        comments: '2.6.2 The title, history, institution, source, references, and comment attributes are all type string'
        

Within each of the elements, validation items can be defined. This items are composed of criteria and/or schema.
criteria represent a selector, matching elements of the given type will be filtered based on the contents. Once this elements are selected, the schema will be applied to each of them, raising errors if not all the selected elements comply with it. This structure allows for conditional validation (i.e.: if variable has attribute X it also needs attribute Y).
If criteria is not present, schema will be applied to all the elements in the file.
If schema is not present, only count check will be applied.

Comparators

The current validation model provides many ad-hoc comparators to validate complex relationships on a netCDF model, particularly in the context of CF. Most are self-explanatory but there are others that require explanation:

Parse

- criteria:
    name: coordinates
  schema:
    name: coordinates
    type: str
    value:
      parse:
        re: '\S+'
        comparator:
          find: 
            variable:
              name: $VALUE
  comments: "5 The type of the coordinates attribute is a string whose value is a blank separated list of variable names. All specified variable names must exist in the file."

parse can be applied to any string value. It will split the string according to the provided regular expression and for each result, apply a given comparator. In the example above, the contents of the coordinates attribute will be split into words.

Find

The find operator will tokenize the left-side value that is applied to and use it to find an element, using said token for any of the fields. The example above uses the each parse entry and looks for a variable which name matches it.

udunits

- criteria:
    connector: OR
    statements:
      - attributes:
          - name: units
            value:
              udunits:
                is_reftime: true

      - attributes:
          - name: axis
            value: T

      - attributes:
          - name: standard_name
            value: 
              choice:
                - time
                - forecast_reference_time
  schema:
    attributes:
      - name: units
        value:
          udunits:
            is_reftime: true
  comments: '4.3.3 The time units of a time coordinate variable must contain a reference date/time.'

udunits operator allows for easy checks against Unidata UDUNITS package.

Locator

      - criteria:
          dimensions_count: 1
          dimensions:
            - name:
                locator: 
                  path: ./
                  type: variable
                  attr: name
        schema:
          value:
            monotonic: True
        comments: '5 A coordinate variable must have values that are strictly monotonic (increasing or decreasing).'

A locator allows to retrieve a value relative to the matched element. In the example above, we retrieve the name of the variable to check it against the dimension name.

What's next

The current goal is to asses if the validation model requires any additional features to fully allow for the implementation of CF-11 as an schema.

My intention with this post is to open the discussion for further improvements or feature request, as well as tapping into your CF expertise to identify potential pitfalls when implementing CF as an schema with the current model.

Once the validation model has been consolidated (I'm considering a soft deadline of end of October), I will start with the implementation of the tool to actually run the validations, aiming for 1.0.0 release before the end of the year.

[gtesoro]

Description updated