Skip to content

Commit

Permalink
Docs.
Browse files Browse the repository at this point in the history
  • Loading branch information
david-i-berry committed Feb 2, 2024
1 parent 1a5b2d7 commit 37039d4
Show file tree
Hide file tree
Showing 3 changed files with 161 additions and 3 deletions.
9 changes: 9 additions & 0 deletions docs/source/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,12 @@ The following output should be shown:
data data workflows
mappings stored mappings
Environment variables
*********************
Three environment variables are defined and can be used to control the originator centre and
sub centre in the generated BUFR files and the system search path used to fin dBUFR mapping templates,
see :ref:`BUFR template page <mapping>`

- BUFR_ORIGINATING_CENTRE - Specifies the originating centre of the BUFR data, Common Code Table C-11
- BUFR_ORIGINATING_SUBCENTRE - Specifies the originating sub centre of the BUFR data, Common Code Table C-12
- CSV2BUFR_TEMPLATES - Path to search for BUFR templates
17 changes: 14 additions & 3 deletions docs/source/mapping.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ BUFR template mapping
=====================

The mapping between the input CSV data and the output BUFR data is specified in a JSON file.
The csv2bufr module validates the mapping file against the schema shown at the bottom of this page prior to attempted the transformation to BUFR.
The csv2bufr module validates the mapping file against the schema shown at the bottom of this page prior to attempting the transformation to BUFR.
This schema specifies 7 primary properties all of which are mandatory:

- ``inputDelayedDescriptorReplicationFactor`` - array of integers, values for the delayed descriptor replication factors to use
Expand Down Expand Up @@ -66,7 +66,7 @@ Similarly the keys for the different data elements can be found at:
- `<https://confluence.ecmwf.int/display/ECC/WMO%3D37+element+table>`_

input<Short|Extended>DelayedDescriptorReplicationFactor
---------------------------------------
-------------------------------------------------------
Due to the way that eccodes works any delayed replication factors need to be specified before encoding and included in the mapping file.
This currently limits the use of the delayed replication factors to static values for a given mapping. For example
every data file that uses a given mapping file has the same optional elements present or the same number of levels in an
Expand Down Expand Up @@ -196,4 +196,15 @@ the ``scale`` and ``offset`` fields. Some additional examples are given below.
Schema
------

.. literalinclude:: ../../csv2bufr/resources/mappings/mapping_schema.json
.. literalinclude:: ../../csv2bufr/resources/mappings/csv2bufr-template-v2.json

Built in templates and search path
----------------------------------

Several preconfigured templates are available from the csv2bufr-templates repository:

- https://github.com/wmo-im/csv2bufr-templates

By default, ``csv2bufr`` searches in the current working directory and the
``/opt/csv2bufr/templates`` directory, additional search paths can be added by setting
the ``CSV2BUFR_TEMPLATES`` environment variable.
138 changes: 138 additions & 0 deletions docs/source/resources/csv2bufr-template-v2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
{
"$id": "csv2bufr.wis2.0.node.wis",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"conformsTo": {},
"metadata": {
"type": "object",
"requiered": ["label","description","version","author","editor","dateCreated","dateModified","id"],
"properties": {
"label": {
"type": "string"
},
"description": {
"type": "string"
},
"version": {
"type": "string"
},
"author": {
"type": "string"
},
"editor": {
"type": "string"
},
"dateCreated": {
"type": "string",
"format": "date"
},
"dateModified": {
"type": "string",
"format": "date"
},
"id": {
"type": "string",
"format": "uuid4"
}
}
},

"inputShortDelayedDescriptorReplicationFactor": {
"type": "array",
"items": {"type": "integer"}
},
"inputDelayedDescriptorReplicationFactor": {
"type": "array",
"items": {"type": "integer"}
},
"inputExtendedDelayedDescriptorReplicationFactor": {
"type": "array",
"items": {"type": "integer"}
},
"number_header_rows": {
"type": "integer",
"description": "Number of header rows in file before the data"
},
"column_names_row": {
"type": "integer",
"description": "Which header line the column names is given on"

},
"wigos_station_identifier": {
"type": "string",
"description": "Either the WIGOS station identifier for the data or the column in the CSV file containing the identifier"
},
"delimiter": {
"type": "string",
"description": "The delimiter used to separate fields in the input csv file, must be one of ',', ';'. '|' or [tab]"
},
"quoting": {
"type": "string",
"description": "CSV quoting method to use, must be one of QUOTE_NONNUMERIC, QUOTE_ALL, QUOTE_MINIMAL or QUOTE_NONE"
},
"quotechar": {
"type": "string",
"description": "quote character to use, e.g. \", ' etc"
},
"header":{
"type": "array",
"items": {"$ref": "#/$defs/bufr_element"},
"description": "Contents of header sections of BUFR message"
},
"data": {
"type": "array",
"items": {"$ref": "#/$defs/bufr_element"},
"description": "mapping from CSV file (or metadata json file) to BUFR"
}
},
"required" : [
"conformsTo", "metadata",
"inputShortDelayedDescriptorReplicationFactor",
"inputDelayedDescriptorReplicationFactor",
"inputExtendedDelayedDescriptorReplicationFactor",
"column_names_row","number_header_rows","header","data"],

"$defs":{
"bufr_element": {
"type": "object",
"properties": {
"eccodes_key": {
"type": "string",
"descripition": "eccodes key used to set the value in the BUFR data"
},
"value": {
"type": [
"string"
],
"description": "where to extract the value from, can be one off 'data','metadata','const','array' followed by the value or column header"
},
"valid_min": {
"type": "string",
"description": "Minimum valid value for parameter if set"
},
"valid_max": {
"type": "string",
"description": "Maximum value for for the parameter if set"
},
"scale": {
"type": "string",
"description": "Value used to scale the data by before encoding using the same conventions as in BUFR"
},
"offset": {
"type": "string",
"description": "Value added to the data before encoding to BUFR following the same conventions as BUFR"
}
},
"required": ["eccodes_key", "value"],
"allOf": [
{
"dependentRequired": {"scale": ["offset"]}
},
{
"dependentRequired": {"offset": ["scale"]}
}
]
}
}
}

0 comments on commit 37039d4

Please sign in to comment.