diff --git a/CHANGELOG.md b/CHANGELOG.md index 179828d..b007185 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - split dataset creation and storage to zarr into separate functions `mllam_data_prep.create_dataset(...)` and `mllam_data_prep.create_dataset_zarr(...)` respectively ![\#7](https://github.com/mllam/mllam-data-prep/pull/7) +- changes to spec from v0.1.0: + - `sampling_dim` removed from `architectures` section of spec, this is not needed to create the training data + - selection on variable coordinates values is now set with `inputs.{dataset_name}.variables.{variable_name}.values` + rather than `inputs.{dataset_name}.variables.{variable_name}.sel` + - when dimension-mapping method `stack_variables_by_var_name` is used the formatting string for the new variable + is now called `name_format` rather than `name` + - when dimension-mapping is done by simply renaming a dimension this configuration now needs to be set by providing + the named method (`rename`) explicitly through the `method` key, i.e. rather than `{to_dim}: {from_dim}` it is now + `{to_dim}: {method: rename, dim: {from_dim}}` to match the signature of the other dimension-mapping methods. + - coordinate value ranges for the dimensions that the architecture expects as input has been renamed from + `architecture.input_ranges` to `architecture.input_coord_ranges` to make the use more clear + - attribute `inputs.{dataset_name}.name` attribute has been removed, with the key `dataset_name` this is + superfluous + ## [v0.1.0](https://github.com/mllam/mllam-data-prep/releases/tag/v0.1.0) First tagged release of `mllam-data-prep` which includes functionality to diff --git a/README.md b/README.md index 0700ae0..79e5fa4 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ A training dataset is constructed by declaring in a yaml configuration file (for The configuration is principally a means to represent how the dimensions of a given variable in a source dataset should be mapped to the dimensions and input variables of the model architecture to be trained. -The full configuration file specification is given in [mllam_data_prep/config/spec.py](mllam_data_prep/config/spec.py). +The configuration is given in yaml-format and the file specification is defined using python3 [dataclasses](https://docs.python.org/3/library/dataclasses.html) (serialised to yaml using [dataclasses-wizard](https://dataclass-wizard.readthedocs.io/en/latest/)) and defined in [mllam_data_prep/config.py](mllam_data_prep/config.py). ## Installation @@ -55,3 +55,174 @@ python -m mllam_data_prep example.danra.yaml Example output: ![](docs/example_output.png) + +## Configuration file + +A full example configuration file is given in [example.danra.yaml](example.danra.yaml), and reproduced here for completeness: + +```yaml +schema_version: v0.2.0 +dataset_version: v0.1.0 + +architecture: + input_variables: + static: [grid_index, static_feature] + state: [time, grid_index, state_feature] + forcing: [time, grid_index, forcing_feature] + input_coord_ranges: + time: + start: 1990-09-03T00:00 + end: 1990-09-09T00:00 + step: PT3H + chunking: + time: 1 + +inputs: + danra_height_levels: + path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/height_levels.zarr + dims: [time, x, y, altitude] + variables: + u: + altitude: + values: [100,] + units: m + v: + altitude: + values: [100, ] + units: m + dim_mapping: + time: + method: rename + dim: time + state_feature: + method: stack_variables_by_var_name + dims: [altitude] + name_format: f"{var_name}{altitude}m" + grid_index: + method: stack + dims: [x, y] + target_architecture_variable: state + + danra_surface: + path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/single_levels.zarr + dims: [time, x, y] + variables: + # shouldn't really be using sea-surface pressure as "forcing", but don't + # have radiation varibles in danra yet + - pres_seasurface + dim_mapping: + time: + method: rename + dim: time + grid_index: + method: stack + dims: [x, y] + forcing_feature: + method: stack_variables_by_var_name + name_format: f"{var_name}" + target_architecture_variable: forcing + + danra_lsm: + path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/lsm.zarr + dims: [x, y] + variables: + - lsm + dim_mapping: + grid_index: + method: stack + dims: [x, y] + static_feature: + method: stack_variables_by_var_name + name_format: f"{var_name}" + target_architecture_variable: static +``` + +Apart from identifies to keep track of the configuration file format version and the datasets version, the configuration file is divided into two main sections: + +- `architecture`: defines the input variables and dimensions of the model architecture to be trained. These are the variables and dimensions that the inputs datasets will be mapped to. +- `inputs`: a list of source datasets to extract data from. These are the datasets that will be mapped to the architecture defined in the `architecture` section. + +### The `architecture` section + +```yaml +architecture: + input_variables: + static: [grid_index, static_feature] + state: [time, grid_index, state_feature] + forcing: [time, grid_index, forcing_feature] + input_coord_ranges: + time: + start: 1990-09-03T00:00 + end: 1990-09-09T00:00 + step: PT3H + chunking: + time: 1 +``` + +The `architecture` section defines three things: + +1. `input_variables`: what input variables the model architecture you are targeting expects, and what the dimensions are for each of these variables. +2. `input_coord_ranges`: the range of values for each of the dimensions that the model architecture expects as input. These are optional, but allows you to ensure that the training dataset is created with the correct range of values for each dimension. +3. `chunking`: the chunk sizes to use when writing the training dataset to zarr. This is optional, but can be used to optimise the performance of the zarr dataset. By default the chunk sizes are set to the size of the dimension, but this can be overridden by setting the chunk size in the configuration file. A common choice is to set the dimension along which you are batching to align with the of each training item (e.g. if you are training a model with time-step roll-out of 10 timesteps, you might choose a chunksize of 10 along the time dimension). + +### The `inputs` section + +```yaml +inputs: + danra_height_levels: + path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/height_levels.zarr + dims: [time, x, y, altitude] + variables: + u: + altitude: + values: [100,] + units: m + v: + altitude: + values: [100, ] + units: m + dim_mapping: + time: + method: rename + dim: time + state_feature: + method: stack_variables_by_var_name + dims: [altitude] + name_format: f"{var_name}{altitude}m" + grid_index: + method: stack + dims: [x, y] + target_architecture_variable: state + + danra_surface: + path: https://mllam-test-data.s3.eu-north-1.amazonaws.com/single_levels.zarr + dims: [time, x, y] + variables: + # shouldn't really be using sea-surface pressure as "forcing", but don't + # have radiation varibles in danra yet + - pres_seasurface + dim_mapping: + time: + method: rename + dim: time + grid_index: + method: stack + dims: [x, y] + forcing_feature: + method: stack_variables_by_var_name + name_format: f"{var_name}" + target_architecture_variable: forcing + + ... +``` + +The `inputs` section defines the source datasets to extract data from. Each source dataset is defined by a key (e.g. `danra_height_levels`) which names the source, and the attributes of the source dataset: + +- `path`: the path to the source dataset. This can be a local path or a URL to e.g. a zarr dataset or netCDF file, anything that can be read by `xarray.open_dataset(...)`. +- `dims`: the dimensions that the source dataset is expected to have. This is used to check that the source dataset has the expected dimensions and also makes it clearer in the config file what the dimensions of the source dataset are. +- `variables`: selects which variables to extract from the source dataset. This may either be a list of variable names, or a dictionary where each key is the variable name and the value defines a dictionary of coordinates to do selection on. When doing selection you may also optionally define the units of the variable to check that the units of the variable match the units of the variable in the model architecture. +- `target_architecture_variable`: the variable in the model architecture that the source dataset should be mapped to. +- `dim_mapping`: defines how the dimensions of the source dataset should be mapped to the dimensions of the model architecture. This is done by defining a method to apply to each dimension. The methods are: + - `rename`: simply rename the dimension to the new name + - `stack`: stack the listed dimension to create the dimension in the output + - `stack_variables_by_var_name`: stack the dimension into the new dimension, and also stack the variable name into the new variable name. This is useful when you have multiple variables with the same dimensions that you want to stack into a single variable. diff --git a/example.danra.yaml b/example.danra.yaml index b0a50b4..2f1cfdd 100644 --- a/example.danra.yaml +++ b/example.danra.yaml @@ -1,13 +1,12 @@ -schema_version: v0.1.0 +schema_version: v0.2.0 dataset_version: v0.1.0 architecture: - sampling_dim: time input_variables: static: [grid_index, static_feature] state: [time, grid_index, state_feature] forcing: [time, grid_index, forcing_feature] - input_range: + input_coord_ranges: time: start: 1990-09-03T00:00 end: 1990-09-09T00:00 @@ -22,20 +21,22 @@ inputs: variables: u: altitude: - sel: [100, ] + values: [100,] units: m v: altitude: - sel: [100, ] + values: [100, ] units: m dim_mapping: - time: time + time: + method: rename + dim: time state_feature: method: stack_variables_by_var_name dims: [altitude] name_format: f"{var_name}{altitude}m" grid_index: - method: flatten + method: stack dims: [x, y] target_architecture_variable: state @@ -47,9 +48,11 @@ inputs: # have radiation varibles in danra yet - pres_seasurface dim_mapping: - time: time + time: + method: rename + dim: time grid_index: - method: flatten + method: stack dims: [x, y] forcing_feature: method: stack_variables_by_var_name @@ -63,7 +66,7 @@ inputs: - lsm dim_mapping: grid_index: - method: flatten + method: stack dims: [x, y] static_feature: method: stack_variables_by_var_name diff --git a/example.yaml b/example.yaml deleted file mode 100644 index df3a39c..0000000 --- a/example.yaml +++ /dev/null @@ -1,132 +0,0 @@ -schema_version: v0.1.0 -dataset_version: v0.1.0 - -architecture: - sampling_dim: time - input_variables: - static: [grid_index, feature] - state: [time, grid_index, feature] - forcing: [time, grid_index, feature] - input_range: - time: - start: 2000-01-01T00:00 - end: 2001-01-01T00:00 - step: PT1H - -inputs: - danra_height_levels: - path: /data/danra/height_levels.zarr - attributes: - version: v0.3.0 - dims: [analysis_time, x, y, altitude] - variables: - u: - altitude: - sel: [50, 100, 200, 300, 500, 700, 850, 1000] - units: m - v: - altitude: - sel: [50, 100, 200, 300, 500, 700, 850, 1000] - units: m - t: - altitude: - sel: [50, 100, 200, 300, 500, 700, 850, 1000] - units: m - dim_mapping: - time: analysis_time - feature: - method: stack_variables_by_var_name - dims: [altitude] - name_format: f"{var_name}_{altitude}" - grid_index: - method: flatten - dims: [x, y] - target_architecture_variable: state - - danra_pressure_levels: - path: /data/danra/pressure_levels.zarr - attributes: - version: v0.3.0 - dims: [analysis_time, x, y, pressure] - variables: - u: - pressure: - sel: [1000, 850, 700, 500, 300, 200, 100] - units: hPa - dim_mapping: - time: analysis_time - feature: - method: stack_variables_by_var_name - dims: [pressure] - name_format: f"{var_name}_{pressure}" - grid_index: x, y - target_architecture_variable: state - - danra_single_levels: - path: /data/danra/single_levels.zarr - attributes: - version: v0.3.0 - dims: [analysis_time, x, y] - variables: u10m, v10m, t2m - dim_mapping: - time: analysis_time - feature: - method: stack_variables_by_var_name - name_format: f"{var_name}" - grid_index: - method: flatten - dims: [x, y] - target_architecture_variable: state - - danra_single_levels_forcings: - path: /data/danra/single_levels.zarr - attributes: - version: v0.3.0 - dims: [analysis_time, x, y] - variables: nswlr - dim_mapping: - time: analysis_time - feature: - method: stack_variables_by_var_name - name_format: f"{var_name}" - grid_index: - method: flatten - dims: [x, y] - target_architecture_variable: forcing - - danra_static2d: - path: /data/danra/static2d.zarr - attributes: - version: v0.3.0 - dims: [x, y] - variables: [topography_height, land_area_fraction] - dim_mapping: - grid_index: - method: flatten - dims: [x, y] - target_architecture_variable: static - - meps_ensemble_forecasts: - path: /data/meps/ensemble_forecasts.zarr - variables: [u, v, t] - dims: [analysis_time, forecast_time, ensemble_member, x, y] - dim_mapping: - time: forecast_time - grid_index: x, y - sub_sampling: - analysis_time: - time: 0 - ensemble_member: "random" - target_architecture_variable: state - - dini_forecast: - path: /data/dini_forecasts_2000_2010.zarr - variables: [u, v, t] - dims: [analysis_time, forecast_time, x, y] - dim_mapping: - time: forecast_time - grid_index: x, y - sub_sampling: - analysis_time: - time: 0 - target_architecture_variable: state diff --git a/mllam_data_prep/__init__.py b/mllam_data_prep/__init__.py index 44dd160..5de1755 100644 --- a/mllam_data_prep/__init__.py +++ b/mllam_data_prep/__init__.py @@ -1,2 +1,3 @@ # expose the public API +from .config import Config, InvalidConfigException # noqa from .create_dataset import create_dataset, create_dataset_zarr # noqa diff --git a/mllam_data_prep/config.py b/mllam_data_prep/config.py new file mode 100644 index 0000000..68d2737 --- /dev/null +++ b/mllam_data_prep/config.py @@ -0,0 +1,251 @@ +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Union + +import dataclass_wizard +from dataclass_wizard import JSONWizard + + +class InvalidConfigException(Exception): + pass + + +class GlobalJSONMeta(JSONWizard.Meta): + """ + Global settings for the JSON load/dump process, that should apply to + *all* subclasses of `JSONWizard`. + + Note: it does not matter where this class is defined, as long as it's + declared before any methods in `JSONWizard` are called. + """ + + raise_on_unknown_json_key = True + + +@dataclass +class Range: + """ + Defines a range for a variable to be used for selection, i.e. + `xarray.Dataset.sel({var_name}: slice({start}, {end}, {step}))`, the variable + name is the key in the dictionary and the slice object is created from the + `start`, `end`, and `step` attributes. + + Attributes + ---------- + start: str + The start of the range, e.g. "1990-09-03T00:00", 0, or 0.0. + end: str + The end of the range, e.g. "1990-09-04T00:00", 1, or 1.0. + step: str + The step size for the range, e.g. "PT3H", 1, or 1.0. If not given + then the entire range will be selected. + """ + + start: Union[str, int, float] + end: Union[str, int, float] + step: Union[str, int, float] = None + + +@dataclass +class ValueSelection: + """ + Defines a selection on the coordinate values of a variable, the + `values` attribute can either be a list of values to select or a + `Range` object to select a range of values. This is used to create + a slice object for the selection. Optionally, the `units` attribute can be + used to specify the units of the values which will used to ensure that + the `units` attribute of the variable has the same value. + + Attributes: + values: The values to select. + units: The units of the values. + """ + + values: Union[List[Union[float, int]], Range] + units: str = None + + +@dataclass +class DimMapping: + """ + Defines the process for mapping dimensions and variables from an input + dataset to a single new dimension (as in dimension in the + output dataset of the dataset generation). + + There are three methods implemented for mapping: + - "rename": + Renames a dimension in the dataset to a new name. + + E.g. adding a dim-mapping as `{"time": {"method": "rename", "dim": "analysis_time"}}` + will rename the "analysis_time" dimension in the input dataset to "time" dimension in the output. + + - "stack_variables_by_var_name": + Stacks all variables along a new dimension that is mapped to the output dimensions name given. + + E.g. adding a dim-mapping as + `{"state_feature": {"method": "stack_variables_by_var_name", "name_format": "{var_name}{altitude}m", dims: [altitude]}}` + will stack all variables in the input dataset along the "state_feature" dimension in the output + and the coordinate values will be given as f"{var_name}{altitude}m" where `var_name` is the name + of the variable and `altitude` is the value of the "altitude" coordinate. + If any dimensions are specified in the `dims` attribute, then the these dimensions will + also be stacked into this new dimension, and the `name_format` attribute can be used to + use the coordinate values from the stacked dimensions in the new coordinate values. + + - "stack": + Stacks the provided coordinates and maps the result to the output dimension. + + E.g. `{"grid_index": {"method": "stack", "dims": ["x", "y"]}}` will stack the "x" and "y" + dimensions in the input dataset into a new "grid_index" dimension in the output. + + Attributes: + method: The method used for mapping. + dims: The dimensions to be mapped. + name_format: The format for naming the mapped dimensions. + + Attributes + ---------- + method: str + The method used for mapping. The options are: + - "rename": Renames a dimension in the dataset to a new name. + - "stack_variables_by_var_name": Stacks all variables along a new dimension that is mapped to the output dimensions name given. + - "stack": Stacks the provided coordinates and maps the result to the output dimension. + dims: List[str] + The dimensions to be mapped when using the "stack" or "stack_variables_by_var_name" methods. + dim: str + The dimension to be renamed when using the "rename" method. + name_format: str + The format for naming the mapped dimensions when using the "stack_variables_by_var_name" method. + """ + + method: str + dims: Optional[List[str]] = None + dim: Optional[str] = None + name_format: str = field(default=None) + + +@dataclass +class InputDataset: + """ + Definition of a single input dataset which will be mapped to one the + variables that have been defined as input variables for model architecture + being targeted by the dataset. + The definition for a input dataset includes setting + 1) the path to the dataset, + 2) the expected dimensions of the dataset, + 3) the variables to select from the dataset (and optionally subsection + along the coordinates for each variable) and finally + 4) the method by which the dimensions and variables of the dataset are + mapped to the dimensions of the architecture's input variables (this + includes stacking of all the selected variables into a new single + variable along a new coordinate, and may include renaming and + stacking dimensions existing dimensions). + + Attributes + ---------- + path: str + Path to the dataset, e.g. the path to a zarr dataset or netCDF file. + This can be anything that can be passed to `xarray.open_dataset` + dims: List[str] + List of the expected dimensions of the dataset. E.g. `["time", "x", "y"]`. + These will be checked to ensure consistency of the dataset being read. + variables: Union[List[str], Dict[str, Dict[str, ValueSelection]]] + List of the variables to select from the dataset. E.g. `["temperature", "precipitation"]` + or a dictionary where the keys are the variable names and the values are dictionaries + defining the selection for each variable. E.g. `{"temperature": levels: {"values": [1000, 950, 900]}}` + would select the "temperature" variable and only the levels 1000, 950, and 900. + dim_mapping: Dict[str, DimMapping] + Mapping of the variables and dimensions in the input dataset to the dimensions of the + architecture's input variables. The key is the name of the architecture dimension to map to + and the ´DimMapping´ describes how to map the dimensions and variables of the input dataset + to this input dimension for the architecture. + target_architecture_variable: str + The name of the variable in the architecture that this dataset is intended to map to. If multiple + datasets map to the same variable, then the data from all datasets will be concatenated along the + dimension that isn't shared (e.g. two datasets that coincide in space and time will only differ + in the feature dimension, so the two will be combined by concatenating along the feature dimension). + If a single shared coordinate cannot be found then an exception will be raised. + """ + + path: str + dims: List[str] + variables: Union[List[str], Dict[str, Dict[str, ValueSelection]]] + dim_mapping: Dict[str, DimMapping] + target_architecture_variable: str + attributes: Dict[str, Any] = None + target_architecture_variable: str + + +@dataclass +class Architecture: + """ + Information about the model architecture this dataset is intended for. This + covers defining what input variables the architecture expects (and the dimensions of each), + the expected value range for each coordinate, and the chunking information for each dimension. + + Attributes + ---------- + input_variables: Dict[str, List[str]] + Defines the input variables for model architecture. The keys are the + variable names to create and the values are lists of the dimensions. E.g. + `{"static": ["grid_index", "feature"], "state": ["time", "grid_index", "state_feature"]}`. + would define that the architecture expects a variable named "static" with + dimensions "grid_index" and "feature" and a variable named "state" with + dimensions "time", "grid_index", and "state_feature". + + input_coord_ranges: Dict[str, Range] + Defines the expected value range for each coordinate. The keys are the + name of the coordinate and the values are the range, e.g. + `{"time": {"start": "1990-09-03T00:00", "end": "1990-09-04T00:00", "step": "PT3H"}}` + would define that the "time" coordinate should have values between + "1990-09-03T00:00" and "1990-09-04T00:00" with a step size of 3 hours. + These range definitions are both used to ensure that the input dataset + has the expected range and to select the correct values from the input + dataset. If not given then the entire range will be selected. + + chunking: Dict[str, int] + Defines the chunking information for each dimension. The keys are the + names of the dimensions and the values are the chunk size for that dimension. + If chunking is not specified for a dimension, then the entire dimension + will be a single chunk. + """ + + input_variables: Dict[str, List[str]] + input_coord_ranges: Dict[str, Range] = None + chunking: Dict[str, int] = None + + +@dataclass +class Config(dataclass_wizard.YAMLWizard): + """Configuration for the model. + + Attributes: + schema_version: Version of the config file schema. + dataset_version: Version of the dataset itself. + architecture: Information about the model architecture this dataset is intended for. + inputs: Input datasets for the model. + + Attributes + ---------- + architecture: Architecture + Information about the model architecture this dataset is intended for. This + covers defining what input variables the architecture expects (and the dimensions of each), + the expected value range for each coordinate, and the chunking information for each dimension. + inputs: Dict[str, InputDataset] + Input datasets for the model. The keys are the names of the datasets and the values are + the input dataset configurations. + schema_version: str + Version string for the config file schema. + dataset_version: str + Version string for the dataset itself. + """ + + architecture: Architecture + inputs: Dict[str, InputDataset] + schema_version: str + dataset_version: str + + +if __name__ == "__main__": + config = Config.from_yaml_file("example.danra.yaml") + import rich + + rich.print(config) diff --git a/mllam_data_prep/config/__init__.py b/mllam_data_prep/config/__init__.py deleted file mode 100644 index 986b595..0000000 --- a/mllam_data_prep/config/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -from .storage import ( - ConfigDict, - InvalidConfigException, - InvalidConfigVariableException, - MissingConfigVariableException, -) diff --git a/mllam_data_prep/config/spec.py b/mllam_data_prep/config/spec.py deleted file mode 100644 index 6c6ae48..0000000 --- a/mllam_data_prep/config/spec.py +++ /dev/null @@ -1,57 +0,0 @@ -FIELD_DESCRIPTIONS = dict( - schema_version="version of the config file schema", - dataset_version="version of the dataset itself", - architecture=dict( - _doc_="Information about the model architecture this dataset is intended for", - sampling_dim="dimension to sample along", - input_variables=dict( - _doc_="Variables the architecture expects as input", - _values_="Dimensions for each input variable (e.g. [time, x, y])", - ), - input_range=dict( - _doc_="Value range for the coordinates that span the input variables (e.g. time)", - _values_=dict( - _doc_="Value range for the coordinate", - start="start of the range", - end="end of the range", - step="step size of the range", - ), - ), - chunking=dict( - _doc_="Chunking for dimensions of the input variables", - _values_=dict( - _doc_="Chunking for a single input variable dimension", - ), - ), - ), - inputs=dict( - _doc_="Input datasets for the model", - _values_=dict( - _doc_="single input dataset", - path="path to the dataset", - dims="dimensions of the dataset", - attributes="attributes the dataset should have", - variables=dict( - _doc_="variables to select from the dataset", - _values_=dict( - _doc_="coordinate selections for the variable", - _values_=dict( - _doc_="selection along a single coordinate", - sel="coordinate values to select from the variable", - units="units of the variable", - ), - ), - ), - dim_mapping=dict( - _doc_="mapping of the dimensions in the dataset to the dimensions of the architecture's input variables", - _values_=dict( - _doc_="mapping of the dimensions in the input dataset to the architecture's input variables", - method="method by which mapping is done (e.g. 'flatten', 'stack_variables_by_var_name')", - dims="dimensions in the source dataset to map from", - name_format="string-format for mapped variable (used when stacking variables to coordinate values)", - ), - ), - target_architecture_variable="name of variable in the output that the given input should stored in", - ), - ), -) diff --git a/mllam_data_prep/config/storage.py b/mllam_data_prep/config/storage.py deleted file mode 100644 index 08ced98..0000000 --- a/mllam_data_prep/config/storage.py +++ /dev/null @@ -1,110 +0,0 @@ -import yaml - -from .spec import FIELD_DESCRIPTIONS - - -class InvalidConfigException(Exception): - """Raised for general config errors""" - - pass - - -class InvalidConfigVariableException(Exception): - """Raised when a config variable is present but is not part of the config spec""" - - pass - - -class MissingConfigVariableException(Exception): - """Raised when a config variable is missing from the config file that is required""" - - pass - - -def _get_nested_from_spec(data: dict = FIELD_DESCRIPTIONS, path=[]): - """ - Get a nested value from a dictionary using a list of keys - allowing for a nested structure to be traversed. A special value named `_values_` is used to indicate - that any key can be used at this level of the dictionary. - """ - key = path[0] - if key in data: - children = data[key] - elif "_values_" in data: - children = data["_values_"] - else: - raise KeyError(f"Key {key} not found in {data}") - - if len(path) == 1: - return children - else: - return _get_nested_from_spec(children, path=path[1:]) - - -class ConfigDict(dict): - """ - Configuration holder that behaves like a dictionary with two differences: - - 1. All key-fetches are checked against the configuration specification to - check whether a specific configuration variable is part of the spec. - (and to provide inline documentation when a variable in the spec - hasn't been set in the config) - 2. All nested dictionaries are wrapped in `ConfigDict` with a reference to - the parent in the config hierarchy which enables indexing into the right - part of the spec - - """ - - def __init__(self, config_dict, parents=[]): - super().__init__(config_dict) - - self._parents = parents - for k, v in self.items(): - item_parents = parents + [k] - - if isinstance(v, dict): - self[k] = ConfigDict(v, parents=item_parents) - - def __getitem__(self, __key): - found_value = False - try: - value = super().__getitem__(__key) - found_value = True - except KeyError: - pass - - field_path = self._parents + [__key] - path_joined = ".".join(field_path) - try: - field_desc = _get_nested_from_spec(path=field_path) - if "_doc_" in field_desc: - field_desc = field_desc["_doc_"] - - if found_value: - return value - - except KeyError as ex: - if found_value: - raise InvalidConfigVariableException( - f"Although a value for {path_joined} does exist ({value})" - " this parameter shouldn't be included in the configuration" - " as it is not part of the config spec." - ) - else: - raise KeyError(f"Unknown config variable `{path_joined}`") from ex - - raise MissingConfigVariableException( - f"Missing config variable `{path_joined}` ({field_desc})" - ) - - def get(self, key, default=None): - try: - return self[key] - except MissingConfigVariableException: - return default - - @classmethod - def load(cls, fp_config): - with open(fp_config, "r") as fh: - config_dict = yaml.load(fh, Loader=yaml.FullLoader) - return cls(config_dict) diff --git a/mllam_data_prep/create_dataset.py b/mllam_data_prep/create_dataset.py index 256f8cd..25b7f7f 100644 --- a/mllam_data_prep/create_dataset.py +++ b/mllam_data_prep/create_dataset.py @@ -1,10 +1,11 @@ import shutil from collections import defaultdict +from pathlib import Path import xarray as xr from loguru import logger -from .config import ConfigDict, InvalidConfigException +from .config import Config, InvalidConfigException from .ops.loading import load_and_subset_dataset from .ops.mapping import map_dims_and_variables from .ops.selection import select_by_kwargs @@ -35,6 +36,7 @@ def _merge_dataarrays_by_target(dataarrays_by_target): attrs_to_keep = ["source_dataset"] dataarrays = [] for target, das in dataarrays_by_target.items(): + logger.info(f"Merging dataarrays for target variable `{target}`") concat_dim = None for da in das: d = da.attrs.get("variables_mapping_dim", None) @@ -80,7 +82,7 @@ def _merge_dataarrays_by_target(dataarrays_by_target): return ds -def create_dataset(config: ConfigDict): +def create_dataset(config: Config): """ Create a dataset from the input datasets specified in the config file. @@ -94,19 +96,19 @@ def create_dataset(config: ConfigDict): xr.Dataset The dataset created from the input datasets with a variable for each target architecture variable. """ - architecture_config = config["architecture"] - architecture_input_ranges = architecture_config.get("input_range", {}) + architecture_config = config.architecture + architecture_coord_ranges = architecture_config.input_coord_ranges dataarrays_by_target = defaultdict(list) - for dataset_name, input_config in config["inputs"].items(): - path = input_config["path"] - variables = input_config["variables"] - target_arch_var = input_config["target_architecture_variable"] - expected_input_attributes = input_config.get("attributes", {}) - expected_input_var_dims = input_config["dims"] + for dataset_name, input_config in config.inputs.items(): + path = input_config.path + variables = input_config.variables + target_arch_var = input_config.target_architecture_variable + expected_input_attributes = input_config.attributes or {} + expected_input_var_dims = input_config.dims - arch_dims = architecture_config["input_variables"][target_arch_var] + arch_dims = architecture_config.input_variables[target_arch_var] logger.info(f"Loading dataset {dataset_name} from {path}") try: @@ -119,7 +121,7 @@ def create_dataset(config: ConfigDict): dataset_name=dataset_name, ) - dim_mapping = input_config["dim_mapping"] + dim_mapping = input_config.dim_mapping # check that there is an entry for each arch dimension # in the dim_mapping so that we know how to construct the @@ -151,11 +153,11 @@ def create_dataset(config: ConfigDict): da_target.attrs["source_dataset"] = dataset_name # only need to do selection for the coordinates that the input dataset actually has - if architecture_input_ranges is not None: + if architecture_coord_ranges is not None: selection_kwargs = {} for dim in arch_dims: - if dim in architecture_input_ranges: - selection_kwargs[dim] = architecture_input_ranges[dim] + if dim in architecture_coord_ranges: + selection_kwargs[dim] = architecture_coord_ranges[dim] da_target = select_by_kwargs(da_target, **selection_kwargs) dataarrays_by_target[target_arch_var].append(da_target) @@ -167,14 +169,15 @@ def create_dataset(config: ConfigDict): # default to making a single chunk for each dimension if chunksize is not specified # in the config - chunking_config = config["architecture"].get("chunking", {}) + chunking_config = config.architecture.chunking or {} + logger.info(f"Chunking dataset with {chunking_config}") chunks = {d: chunking_config.get(d, int(ds[d].count())) for d in ds.dims} ds = ds.chunk(chunks) return ds -def create_dataset_zarr(fp_config): +def create_dataset_zarr(fp_config, fp_zarr: str = None): """ Create a dataset from the input datasets specified in the config file and write it to a zarr file. The path to the zarr file is the same as the config file, but with the extension changed to '.zarr'. @@ -183,14 +186,26 @@ def create_dataset_zarr(fp_config): ---------- fp_config : Path The path to the configuration file. + fp_zarr : Path, optional + The path to the zarr file to write the dataset to. If not provided, the zarr file will be written + to the same directory as the config file with the extension changed to '.zarr'. """ - config = ConfigDict.load(fp_config=fp_config) + config = Config.from_yaml_file(file=fp_config) + + assert ( + config.schema_version == "v0.2.0" + ), f"Expected schema version v0.2.0, got {config.schema_version}" ds = create_dataset(config=config) - fp_out = fp_config.parent / fp_config.name.replace(".yaml", ".zarr") - if fp_out.exists(): - logger.info(f"Removing existing dataset at {fp_out}") - shutil.rmtree(fp_out) - ds.to_zarr(fp_out) - logger.info(f"Wrote training-ready dataset to {fp_out}") + logger.info("Writing dataset to zarr") + if fp_zarr is None: + fp_zarr = fp_config.parent / fp_config.name.replace(".yaml", ".zarr") + else: + fp_zarr = Path(fp_zarr) + + if fp_zarr.exists(): + logger.info(f"Removing existing dataset at {fp_zarr}") + shutil.rmtree(fp_zarr) + ds.to_zarr(fp_zarr) + logger.info(f"Wrote training-ready dataset to {fp_zarr}") diff --git a/mllam_data_prep/ops/loading.py b/mllam_data_prep/ops/loading.py index fa6b1c3..955fafd 100644 --- a/mllam_data_prep/ops/loading.py +++ b/mllam_data_prep/ops/loading.py @@ -9,21 +9,26 @@ def load_and_subset_dataset(fp, variables): Parameters ---------- fp : str - Filepath to the zarr dataset + Filepath to the source dataset, for example the path to a zarr dataset + or a netCDF file (anything that is supported by `xarray.open_dataset` will work) variables : dict Dictionary with the variables to subset with keys as the variable names and values with entries for each coordinate and coordinate values to extract """ - ds = xr.open_zarr(fp) + try: + ds = xr.open_zarr(fp) + except ValueError: + ds = xr.open_dataset(fp) + ds_subset = xr.Dataset() ds_subset.attrs.update(ds.attrs) if isinstance(variables, dict): for var, coords_to_sample in variables.items(): da = ds[var] for coord, sampling in coords_to_sample.items(): - coord_values = sampling["sel"] + coord_values = sampling.values try: da = da.sel(**{coord: coord_values}) except KeyError as ex: @@ -31,7 +36,7 @@ def load_and_subset_dataset(fp, variables): f"Could not find the all coordinate values `{coord_values}` in " f"coordinate `{coord}` in the dataset" ) from ex - expected_units = sampling.get("units", None) + expected_units = sampling.units coord_units = da[coord].attrs.get("units", None) if coord_units is not None and coord_units != expected_units: raise ValueError( diff --git a/mllam_data_prep/ops/mapping.py b/mllam_data_prep/ops/mapping.py index 542630e..9482ff8 100644 --- a/mllam_data_prep/ops/mapping.py +++ b/mllam_data_prep/ops/mapping.py @@ -29,7 +29,7 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims): the architecture dimension, the 'name' key should be the string format to construct the new coordinate values for the architecture dimension. Exactly one of this type of mapping should be used. - - 'flatten': + - 'stack': used to map variables to the architecture dimension by stacking the along the `dims` provided. - 'rename' (or if the input_dim_map is a string naming the dimension to rename): @@ -63,16 +63,13 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims): dim_mapping = dim_mapping.copy() variable_dim_mappings = {} for arch_dim in list(dim_mapping.keys()): - if ( - isinstance(dim_mapping[arch_dim], dict) - and dim_mapping[arch_dim].get("method") == "stack_variables_by_var_name" - ): + if dim_mapping[arch_dim].method == "stack_variables_by_var_name": variable_dim_mappings[arch_dim] = dim_mapping.pop(arch_dim) if len(variable_dim_mappings) > 1: raise ValueError( "Only one mapping which requires stacking variables" " into a single dataarray is allowed, found ones targeting" - " the following arch dimensions: {list(variable_dim_mappings.keys())}" + f" the following arch dimensions: {list(variable_dim_mappings.keys())}" ) elif len(variable_dim_mappings) == 0: raise Exception( @@ -92,35 +89,20 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims): # handle those mappings that involve just renaming or stacking dimensions for arch_dim, input_dim_map in dim_mapping.items(): - if isinstance(input_dim_map, str): - method = "rename" - else: - method = input_dim_map.get("method") + method = input_dim_map.method if method == "rename": - if isinstance(input_dim_map, str): - source_dim = input_dim_map - else: - source_dims = input_dim_map["dims"] - if len(source_dims) != 1: - raise ValueError( - "Exactly one dimension must be given when mapping with the" - f" `rename` method. Instead got: {source_dims}" - ) - source_dim = source_dims[0] - _check_for_malformed_list_arg(input_dim_map) - # if the input_dims is a string, we assume that it is the - # name of the dimension in the input dataset and we rename - # it to the architecture dimension + source_dim = input_dim_map.dim ds = ds.rename({source_dim: arch_dim}) - elif method == "flatten": - source_dims = input_dim_map["dims"] - # if the input_dims is a list, we assume that it is a list of - # dimensions in the input dataset that we want to stack to create the - # architecture dimension, this is for example used for flatting the - # spatial dimensions into a single dimension representing the grid - # index + elif method == "stack": + source_dims = input_dim_map.dims + # when stacking we assume that the input_dims is a list of dimensions + # in the input dataset that we want to stack to create the architecture + # dimension, this is for example used for flatting the spatial dimensions + # into a single dimension representing the grid index ds = ds.stack({arch_dim: source_dims}).reset_index(arch_dim) + else: + raise NotImplementedError(method) # Finally, we handle the stacking of variables to coordinate values. We # might want to deal with variables that exist on multiple coordinate @@ -129,9 +111,9 @@ def map_dims_and_variables(ds, dim_mapping, expected_input_var_dims): # format to construct the new coordinate values is given in the 'name' key. try: arch_dim, variable_dim_map = variable_dim_mappings.popitem() - dims = variable_dim_map.get("dims", []) + dims = variable_dim_map.dims or [] _check_for_malformed_list_arg(dims) - name_format = variable_dim_map["name_format"] + name_format = variable_dim_map.name_format if len(dims) == 0: da = stack_variables_as_coord_values( ds=ds, name_format=name_format, combined_dim_name=arch_dim diff --git a/mllam_data_prep/ops/selection.py b/mllam_data_prep/ops/selection.py index 9dba78d..b335646 100644 --- a/mllam_data_prep/ops/selection.py +++ b/mllam_data_prep/ops/selection.py @@ -2,6 +2,8 @@ import pandas as pd +from ..config import Range + def _normalize_slice_startstop(s): if isinstance(s, pd.Timestamp): @@ -55,33 +57,35 @@ def select_by_kwargs(ds, **config_dict): for coord, selection in config_dict.items(): if coord not in ds.coords: raise ValueError(f"Coordinate {coord} not found in dataset") - if isinstance(selection, dict): - if "start" not in selection and "end" not in selection: + if isinstance(selection, Range): + if selection.start is None and selection.end is None: raise ValueError( f"Selection for coordinate {coord} must have either 'start' and 'end' given" ) - start = _normalize_slice_startstop(selection.get("start")) - end = _normalize_slice_startstop(selection.get("end")) - step = _normalize_slice_step(selection.get("step")) + sel_start = _normalize_slice_startstop(selection.start) + sel_end = _normalize_slice_startstop(selection.end) + sel_step = _normalize_slice_step(selection.step) - ds_orig = ds + assert sel_start != sel_end, "Start and end cannot be the same" - ds = ds.sel({coord: slice(start, end)}) + # we don't select with the step size for now, but simply check (below) that + # the step size in the data is the same as the requested step size + ds = ds.sel({coord: slice(sel_start, sel_end)}) # check that the start and end are in the data - coord_minmax = ds_orig[coord].min().values, ds_orig[coord].max().values - if start is not None and start not in ds[coord].values: + coord_minmax = ds[coord].min().values, ds[coord].max().values + if sel_start is not None and sel_start not in ds[coord].values: raise ValueError( - f"Provided start value for coordinate {coord} ({start}) is not in the data." + f"Provided start value for coordinate {coord} ({sel_start}) is not in the data." f"Coord {coord} spans [{coord_minmax[0]}, {coord_minmax[1]}]" ) - if end is not None and end not in ds[coord].values: + if sel_end is not None and sel_end not in ds[coord].values: raise ValueError( - f"Provided end value for coordinate {coord} ({end}) is not in the data. " + f"Provided end value for coordinate {coord} ({sel_end}) is not in the data. " f"Coord {coord} spans [{coord_minmax[0]}, {coord_minmax[1]}]" ) - if step is not None: + if sel_step is not None: # check that the step requested is exactly what the data has all_steps = ds[coord].diff(dim=coord).values first_step = ( @@ -92,9 +96,9 @@ def select_by_kwargs(ds, **config_dict): raise ValueError( f"Step size for coordinate {coord} is not constant: {all_steps}" ) - if step != first_step: + if sel_step != first_step: raise ValueError( - f"Step size for coordinate {coord} is not the same as requested: {first_step} != {step}" + f"Step size for coordinate {coord} is not the same as requested: {first_step} != {sel_step}" ) elif isinstance(selection, list): diff --git a/pdm.lock b/pdm.lock index 03c161b..1c98463 100644 --- a/pdm.lock +++ b/pdm.lock @@ -3,15 +3,16 @@ [metadata] groups = ["default", "dev"] -strategy = ["cross_platform"] +strategy = ["cross_platform", "inherit_metadata"] lock_version = "4.4.1" -content_hash = "sha256:f383ea64249a42ea354a36779f3da8116cff518581d1d896386c374a3d443ab3" +content_hash = "sha256:048802da94c917546e1cdfef5da631dc99e180ef375fe28910d00d344ad10081" [[package]] name = "aiohttp" -version = "3.9.3" +version = "3.9.5" requires_python = ">=3.8" summary = "Async http client/server framework (asyncio)" +groups = ["default"] dependencies = [ "aiosignal>=1.1.2", "async-timeout<5.0,>=4.0; python_version < \"3.11\"", @@ -21,52 +22,52 @@ dependencies = [ "yarl<2.0,>=1.0", ] files = [ - {file = "aiohttp-3.9.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:939677b61f9d72a4fa2a042a5eee2a99a24001a67c13da113b2e30396567db54"}, - {file = "aiohttp-3.9.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1f5cd333fcf7590a18334c90f8c9147c837a6ec8a178e88d90a9b96ea03194cc"}, - {file = "aiohttp-3.9.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:82e6aa28dd46374f72093eda8bcd142f7771ee1eb9d1e223ff0fa7177a96b4a5"}, - {file = "aiohttp-3.9.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f56455b0c2c7cc3b0c584815264461d07b177f903a04481dfc33e08a89f0c26b"}, - {file = "aiohttp-3.9.3-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bca77a198bb6e69795ef2f09a5f4c12758487f83f33d63acde5f0d4919815768"}, - {file = "aiohttp-3.9.3-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e083c285857b78ee21a96ba1eb1b5339733c3563f72980728ca2b08b53826ca5"}, - {file = "aiohttp-3.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ab40e6251c3873d86ea9b30a1ac6d7478c09277b32e14745d0d3c6e76e3c7e29"}, - {file = "aiohttp-3.9.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:df822ee7feaaeffb99c1a9e5e608800bd8eda6e5f18f5cfb0dc7eeb2eaa6bbec"}, - {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:acef0899fea7492145d2bbaaaec7b345c87753168589cc7faf0afec9afe9b747"}, - {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:cd73265a9e5ea618014802ab01babf1940cecb90c9762d8b9e7d2cc1e1969ec6"}, - {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:a78ed8a53a1221393d9637c01870248a6f4ea5b214a59a92a36f18151739452c"}, - {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_s390x.whl", hash = "sha256:6b0e029353361f1746bac2e4cc19b32f972ec03f0f943b390c4ab3371840aabf"}, - {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:7cf5c9458e1e90e3c390c2639f1017a0379a99a94fdfad3a1fd966a2874bba52"}, - {file = "aiohttp-3.9.3-cp310-cp310-win32.whl", hash = "sha256:3e59c23c52765951b69ec45ddbbc9403a8761ee6f57253250c6e1536cacc758b"}, - {file = "aiohttp-3.9.3-cp310-cp310-win_amd64.whl", hash = "sha256:055ce4f74b82551678291473f66dc9fb9048a50d8324278751926ff0ae7715e5"}, - {file = "aiohttp-3.9.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6b88f9386ff1ad91ace19d2a1c0225896e28815ee09fc6a8932fded8cda97c3d"}, - {file = "aiohttp-3.9.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c46956ed82961e31557b6857a5ca153c67e5476972e5f7190015018760938da2"}, - {file = "aiohttp-3.9.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:07b837ef0d2f252f96009e9b8435ec1fef68ef8b1461933253d318748ec1acdc"}, - {file = "aiohttp-3.9.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dad46e6f620574b3b4801c68255492e0159d1712271cc99d8bdf35f2043ec266"}, - {file = "aiohttp-3.9.3-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5ed3e046ea7b14938112ccd53d91c1539af3e6679b222f9469981e3dac7ba1ce"}, - {file = "aiohttp-3.9.3-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:039df344b45ae0b34ac885ab5b53940b174530d4dd8a14ed8b0e2155b9dddccb"}, - {file = "aiohttp-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7943c414d3a8d9235f5f15c22ace69787c140c80b718dcd57caaade95f7cd93b"}, - {file = "aiohttp-3.9.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:84871a243359bb42c12728f04d181a389718710129b36b6aad0fc4655a7647d4"}, - {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:5eafe2c065df5401ba06821b9a054d9cb2848867f3c59801b5d07a0be3a380ae"}, - {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:9d3c9b50f19704552f23b4eaea1fc082fdd82c63429a6506446cbd8737823da3"}, - {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:f033d80bc6283092613882dfe40419c6a6a1527e04fc69350e87a9df02bbc283"}, - {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:2c895a656dd7e061b2fd6bb77d971cc38f2afc277229ce7dd3552de8313a483e"}, - {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:1f5a71d25cd8106eab05f8704cd9167b6e5187bcdf8f090a66c6d88b634802b4"}, - {file = "aiohttp-3.9.3-cp311-cp311-win32.whl", hash = "sha256:50fca156d718f8ced687a373f9e140c1bb765ca16e3d6f4fe116e3df7c05b2c5"}, - {file = "aiohttp-3.9.3-cp311-cp311-win_amd64.whl", hash = "sha256:5fe9ce6c09668063b8447f85d43b8d1c4e5d3d7e92c63173e6180b2ac5d46dd8"}, - {file = "aiohttp-3.9.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:38a19bc3b686ad55804ae931012f78f7a534cce165d089a2059f658f6c91fa60"}, - {file = "aiohttp-3.9.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:770d015888c2a598b377bd2f663adfd947d78c0124cfe7b959e1ef39f5b13869"}, - {file = "aiohttp-3.9.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ee43080e75fc92bf36219926c8e6de497f9b247301bbf88c5c7593d931426679"}, - {file = "aiohttp-3.9.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:52df73f14ed99cee84865b95a3d9e044f226320a87af208f068ecc33e0c35b96"}, - {file = "aiohttp-3.9.3-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:dc9b311743a78043b26ffaeeb9715dc360335e5517832f5a8e339f8a43581e4d"}, - {file = "aiohttp-3.9.3-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b955ed993491f1a5da7f92e98d5dad3c1e14dc175f74517c4e610b1f2456fb11"}, - {file = "aiohttp-3.9.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:504b6981675ace64c28bf4a05a508af5cde526e36492c98916127f5a02354d53"}, - {file = "aiohttp-3.9.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a6fe5571784af92b6bc2fda8d1925cccdf24642d49546d3144948a6a1ed58ca5"}, - {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ba39e9c8627edc56544c8628cc180d88605df3892beeb2b94c9bc857774848ca"}, - {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:e5e46b578c0e9db71d04c4b506a2121c0cb371dd89af17a0586ff6769d4c58c1"}, - {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:938a9653e1e0c592053f815f7028e41a3062e902095e5a7dc84617c87267ebd5"}, - {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_s390x.whl", hash = "sha256:c3452ea726c76e92f3b9fae4b34a151981a9ec0a4847a627c43d71a15ac32aa6"}, - {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:ff30218887e62209942f91ac1be902cc80cddb86bf00fbc6783b7a43b2bea26f"}, - {file = "aiohttp-3.9.3-cp312-cp312-win32.whl", hash = "sha256:38f307b41e0bea3294a9a2a87833191e4bcf89bb0365e83a8be3a58b31fb7f38"}, - {file = "aiohttp-3.9.3-cp312-cp312-win_amd64.whl", hash = "sha256:b791a3143681a520c0a17e26ae7465f1b6f99461a28019d1a2f425236e6eedb5"}, - {file = "aiohttp-3.9.3.tar.gz", hash = "sha256:90842933e5d1ff760fae6caca4b2b3edba53ba8f4b71e95dacf2818a2aca06f7"}, + {file = "aiohttp-3.9.5-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:fcde4c397f673fdec23e6b05ebf8d4751314fa7c24f93334bf1f1364c1c69ac7"}, + {file = "aiohttp-3.9.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5d6b3f1fabe465e819aed2c421a6743d8debbde79b6a8600739300630a01bf2c"}, + {file = "aiohttp-3.9.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:6ae79c1bc12c34082d92bf9422764f799aee4746fd7a392db46b7fd357d4a17a"}, + {file = "aiohttp-3.9.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d3ebb9e1316ec74277d19c5f482f98cc65a73ccd5430540d6d11682cd857430"}, + {file = "aiohttp-3.9.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:84dabd95154f43a2ea80deffec9cb44d2e301e38a0c9d331cc4aa0166fe28ae3"}, + {file = "aiohttp-3.9.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c8a02fbeca6f63cb1f0475c799679057fc9268b77075ab7cf3f1c600e81dd46b"}, + {file = "aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c26959ca7b75ff768e2776d8055bf9582a6267e24556bb7f7bd29e677932be72"}, + {file = "aiohttp-3.9.5-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:714d4e5231fed4ba2762ed489b4aec07b2b9953cf4ee31e9871caac895a839c0"}, + {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:e7a6a8354f1b62e15d48e04350f13e726fa08b62c3d7b8401c0a1314f02e3558"}, + {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:c413016880e03e69d166efb5a1a95d40f83d5a3a648d16486592c49ffb76d0db"}, + {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:ff84aeb864e0fac81f676be9f4685f0527b660f1efdc40dcede3c251ef1e867f"}, + {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_s390x.whl", hash = "sha256:ad7f2919d7dac062f24d6f5fe95d401597fbb015a25771f85e692d043c9d7832"}, + {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:702e2c7c187c1a498a4e2b03155d52658fdd6fda882d3d7fbb891a5cf108bb10"}, + {file = "aiohttp-3.9.5-cp310-cp310-win32.whl", hash = "sha256:67c3119f5ddc7261d47163ed86d760ddf0e625cd6246b4ed852e82159617b5fb"}, + {file = "aiohttp-3.9.5-cp310-cp310-win_amd64.whl", hash = "sha256:471f0ef53ccedec9995287f02caf0c068732f026455f07db3f01a46e49d76bbb"}, + {file = "aiohttp-3.9.5-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:e0ae53e33ee7476dd3d1132f932eeb39bf6125083820049d06edcdca4381f342"}, + {file = "aiohttp-3.9.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c088c4d70d21f8ca5c0b8b5403fe84a7bc8e024161febdd4ef04575ef35d474d"}, + {file = "aiohttp-3.9.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:639d0042b7670222f33b0028de6b4e2fad6451462ce7df2af8aee37dcac55424"}, + {file = "aiohttp-3.9.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f26383adb94da5e7fb388d441bf09c61e5e35f455a3217bfd790c6b6bc64b2ee"}, + {file = "aiohttp-3.9.5-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:66331d00fb28dc90aa606d9a54304af76b335ae204d1836f65797d6fe27f1ca2"}, + {file = "aiohttp-3.9.5-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4ff550491f5492ab5ed3533e76b8567f4b37bd2995e780a1f46bca2024223233"}, + {file = "aiohttp-3.9.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f22eb3a6c1080d862befa0a89c380b4dafce29dc6cd56083f630073d102eb595"}, + {file = "aiohttp-3.9.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a81b1143d42b66ffc40a441379387076243ef7b51019204fd3ec36b9f69e77d6"}, + {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:f64fd07515dad67f24b6ea4a66ae2876c01031de91c93075b8093f07c0a2d93d"}, + {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:93e22add827447d2e26d67c9ac0161756007f152fdc5210277d00a85f6c92323"}, + {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:55b39c8684a46e56ef8c8d24faf02de4a2b2ac60d26cee93bc595651ff545de9"}, + {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:4715a9b778f4293b9f8ae7a0a7cef9829f02ff8d6277a39d7f40565c737d3771"}, + {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:afc52b8d969eff14e069a710057d15ab9ac17cd4b6753042c407dcea0e40bf75"}, + {file = "aiohttp-3.9.5-cp311-cp311-win32.whl", hash = "sha256:b3df71da99c98534be076196791adca8819761f0bf6e08e07fd7da25127150d6"}, + {file = "aiohttp-3.9.5-cp311-cp311-win_amd64.whl", hash = "sha256:88e311d98cc0bf45b62fc46c66753a83445f5ab20038bcc1b8a1cc05666f428a"}, + {file = "aiohttp-3.9.5-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:c7a4b7a6cf5b6eb11e109a9755fd4fda7d57395f8c575e166d363b9fc3ec4678"}, + {file = "aiohttp-3.9.5-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:0a158704edf0abcac8ac371fbb54044f3270bdbc93e254a82b6c82be1ef08f3c"}, + {file = "aiohttp-3.9.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d153f652a687a8e95ad367a86a61e8d53d528b0530ef382ec5aaf533140ed00f"}, + {file = "aiohttp-3.9.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:82a6a97d9771cb48ae16979c3a3a9a18b600a8505b1115cfe354dfb2054468b4"}, + {file = "aiohttp-3.9.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:60cdbd56f4cad9f69c35eaac0fbbdf1f77b0ff9456cebd4902f3dd1cf096464c"}, + {file = "aiohttp-3.9.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8676e8fd73141ded15ea586de0b7cda1542960a7b9ad89b2b06428e97125d4fa"}, + {file = "aiohttp-3.9.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:da00da442a0e31f1c69d26d224e1efd3a1ca5bcbf210978a2ca7426dfcae9f58"}, + {file = "aiohttp-3.9.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:18f634d540dd099c262e9f887c8bbacc959847cfe5da7a0e2e1cf3f14dbf2daf"}, + {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:320e8618eda64e19d11bdb3bd04ccc0a816c17eaecb7e4945d01deee2a22f95f"}, + {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:2faa61a904b83142747fc6a6d7ad8fccff898c849123030f8e75d5d967fd4a81"}, + {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:8c64a6dc3fe5db7b1b4d2b5cb84c4f677768bdc340611eca673afb7cf416ef5a"}, + {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_s390x.whl", hash = "sha256:393c7aba2b55559ef7ab791c94b44f7482a07bf7640d17b341b79081f5e5cd1a"}, + {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:c671dc117c2c21a1ca10c116cfcd6e3e44da7fcde37bf83b2be485ab377b25da"}, + {file = "aiohttp-3.9.5-cp312-cp312-win32.whl", hash = "sha256:5a7ee16aab26e76add4afc45e8f8206c95d1d75540f1039b84a03c3b3800dd59"}, + {file = "aiohttp-3.9.5-cp312-cp312-win_amd64.whl", hash = "sha256:5ca51eadbd67045396bc92a4345d1790b7301c14d1848feaac1d6a6c9289e888"}, + {file = "aiohttp-3.9.5.tar.gz", hash = "sha256:edea7d15772ceeb29db4aff55e482d4bcfb6ae160ce144f2682de02f6d693551"}, ] [[package]] @@ -74,6 +75,7 @@ name = "aiosignal" version = "1.3.1" requires_python = ">=3.7" summary = "aiosignal: a list of registered asynchronous callbacks" +groups = ["default"] dependencies = [ "frozenlist>=1.1.0", ] @@ -86,6 +88,7 @@ files = [ name = "asciitree" version = "0.3.3" summary = "Draws ASCII trees." +groups = ["default"] files = [ {file = "asciitree-0.3.3.tar.gz", hash = "sha256:4aa4b9b649f85e3fcb343363d97564aa1fb62e249677f2e18a96765145cc0f6e"}, ] @@ -94,6 +97,8 @@ files = [ name = "asttokens" version = "2.4.1" summary = "Annotate AST trees with source code positions" +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "six>=1.12.0", ] @@ -107,6 +112,8 @@ name = "async-timeout" version = "4.0.3" requires_python = ">=3.7" summary = "Timeout context manager for asyncio programs" +groups = ["default"] +marker = "python_version < \"3.11\"" files = [ {file = "async-timeout-4.0.3.tar.gz", hash = "sha256:4640d96be84d82d02ed59ea2b7105a0f7b33abe8703703cd0ab0bf87c427522f"}, {file = "async_timeout-4.0.3-py3-none-any.whl", hash = "sha256:7405140ff1230c310e51dc27b3145b9092d659ce68ff733fb0cefe3ee42be028"}, @@ -117,6 +124,7 @@ name = "attrs" version = "23.2.0" requires_python = ">=3.7" summary = "Classes Without Boilerplate" +groups = ["default"] files = [ {file = "attrs-23.2.0-py3-none-any.whl", hash = "sha256:99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1"}, {file = "attrs-23.2.0.tar.gz", hash = "sha256:935dc3b529c262f6cf76e50877d35a4bd3c1de194fd41f47a2b7ae8f19971f30"}, @@ -124,12 +132,13 @@ files = [ [[package]] name = "certifi" -version = "2024.2.2" +version = "2024.6.2" requires_python = ">=3.6" summary = "Python package for providing Mozilla's CA Bundle." +groups = ["default"] files = [ - {file = "certifi-2024.2.2-py3-none-any.whl", hash = "sha256:dc383c07b76109f368f6106eee2b593b04a011ea4d55f652c6ca24a754d1cdd1"}, - {file = "certifi-2024.2.2.tar.gz", hash = "sha256:0569859f95fc761b18b45ef421b1290a0f65f147e92a1e5eb3e635f9a5e4e66f"}, + {file = "certifi-2024.6.2-py3-none-any.whl", hash = "sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56"}, + {file = "certifi-2024.6.2.tar.gz", hash = "sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516"}, ] [[package]] @@ -137,6 +146,7 @@ name = "cfgv" version = "3.4.0" requires_python = ">=3.8" summary = "Validate configuration and produce human readable error messages." +groups = ["dev"] files = [ {file = "cfgv-3.4.0-py2.py3-none-any.whl", hash = "sha256:b7265b1f29fd3316bfcd2b330d63d024f2bfd8bcb8b0272f8e19a504856c48f9"}, {file = "cfgv-3.4.0.tar.gz", hash = "sha256:e52591d4c5f5dead8e0f673fb16db7949d2cfb3f7da4582893288f0ded8fe560"}, @@ -147,6 +157,7 @@ name = "charset-normalizer" version = "3.3.2" requires_python = ">=3.7.0" summary = "The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet." +groups = ["default"] files = [ {file = "charset-normalizer-3.3.2.tar.gz", hash = "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5"}, {file = "charset_normalizer-3.3.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:25baf083bf6f6b341f4121c2f3c548875ee6f5339300e08be3f2b2ba1721cdd3"}, @@ -202,6 +213,7 @@ name = "click" version = "8.1.7" requires_python = ">=3.7" summary = "Composable command line interface toolkit" +groups = ["default"] dependencies = [ "colorama; platform_system == \"Windows\"", ] @@ -215,6 +227,7 @@ name = "cloudpickle" version = "3.0.0" requires_python = ">=3.8" summary = "Pickler class to extend the standard pickle.Pickler functionality" +groups = ["default"] files = [ {file = "cloudpickle-3.0.0-py3-none-any.whl", hash = "sha256:246ee7d0c295602a036e86369c77fecda4ab17b506496730f2f576d9016fd9c7"}, {file = "cloudpickle-3.0.0.tar.gz", hash = "sha256:996d9a482c6fb4f33c1a35335cf8afd065d2a56e973270364840712d9131a882"}, @@ -225,6 +238,8 @@ name = "colorama" version = "0.4.6" requires_python = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" summary = "Cross-platform colored terminal text." +groups = ["default", "dev"] +marker = "sys_platform == \"win32\" or platform_system == \"Windows\"" files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, @@ -232,22 +247,33 @@ files = [ [[package]] name = "dask" -version = "2024.2.1" +version = "2024.6.2" requires_python = ">=3.9" summary = "Parallel PyData with Task Scheduling" +groups = ["default"] dependencies = [ "click>=8.1", "cloudpickle>=1.5.0", "fsspec>=2021.09.0", - "importlib-metadata>=4.13.0", + "importlib-metadata>=4.13.0; python_version < \"3.12\"", "packaging>=20.0", "partd>=1.2.0", "pyyaml>=5.3.1", "toolz>=0.10.0", ] files = [ - {file = "dask-2024.2.1-py3-none-any.whl", hash = "sha256:a13fcdeead3bab3576495023f83097adcffe2f03c371c241b5a1f0b232b35b38"}, - {file = "dask-2024.2.1.tar.gz", hash = "sha256:9504a1e9f5d8e5403fae931f9f1660d41f510f48895ccefce856ec6a4c2198d8"}, + {file = "dask-2024.6.2-py3-none-any.whl", hash = "sha256:81b80ee015b2e057b93bb2d1bf13a866136e762e2b24bf54b6b621e8b86b7708"}, + {file = "dask-2024.6.2.tar.gz", hash = "sha256:d429d6b19e85fd1306ac37c188aaf99d03bbe69a6fe59d2b42882b2ac188686f"}, +] + +[[package]] +name = "dataclass-wizard" +version = "0.22.3" +summary = "Marshal dataclasses to/from JSON. Use field properties with initial values. Construct a dataclass schema with JSON input." +groups = ["default"] +files = [ + {file = "dataclass-wizard-0.22.3.tar.gz", hash = "sha256:4c46591782265058f1148cfd1f54a3a91221e63986fdd04c9d59f4ced61f4424"}, + {file = "dataclass_wizard-0.22.3-py2.py3-none-any.whl", hash = "sha256:63751203e54b9b9349212cc185331da73c1adc99c51312575eb73bb5c00c1962"}, ] [[package]] @@ -255,6 +281,8 @@ name = "decorator" version = "5.1.1" requires_python = ">=3.5" summary = "Decorators for Humans" +groups = ["dev"] +marker = "python_version > \"3.6\"" files = [ {file = "decorator-5.1.1-py3-none-any.whl", hash = "sha256:b8c3f85900b9dc423225913c5aace94729fe1fa9763b38939a95226f02d37186"}, {file = "decorator-5.1.1.tar.gz", hash = "sha256:637996211036b6385ef91435e4fae22989472f9d571faba8927ba8253acbc330"}, @@ -264,6 +292,7 @@ files = [ name = "distlib" version = "0.3.8" summary = "Distribution utilities" +groups = ["dev"] files = [ {file = "distlib-0.3.8-py2.py3-none-any.whl", hash = "sha256:034db59a0b96f8ca18035f36290806a9a6e6bd9d1ff91e45a7f172eb17e51784"}, {file = "distlib-0.3.8.tar.gz", hash = "sha256:1530ea13e350031b6312d8580ddb6b27a104275a31106523b8f123787f494f64"}, @@ -271,12 +300,14 @@ files = [ [[package]] name = "exceptiongroup" -version = "1.2.0" +version = "1.2.1" requires_python = ">=3.7" summary = "Backport of PEP 654 (exception groups)" +groups = ["dev"] +marker = "python_version < \"3.11\"" files = [ - {file = "exceptiongroup-1.2.0-py3-none-any.whl", hash = "sha256:4bfd3996ac73b41e9b9628b04e079f193850720ea5945fc96a08633c66912f14"}, - {file = "exceptiongroup-1.2.0.tar.gz", hash = "sha256:91f5c769735f051a4290d52edd0858999b57e5876e9f85937691bd4c9fa3ed68"}, + {file = "exceptiongroup-1.2.1-py3-none-any.whl", hash = "sha256:5258b9ed329c5bbdd31a309f53cbfb0b155341807f6ff7606a1e801a891b29ad"}, + {file = "exceptiongroup-1.2.1.tar.gz", hash = "sha256:a4785e48b045528f5bfe627b6ad554ff32def154f42372786903b7abcfe1aa16"}, ] [[package]] @@ -284,6 +315,8 @@ name = "executing" version = "2.0.1" requires_python = ">=3.5" summary = "Get the currently executing AST node of a frame, and other information" +groups = ["dev"] +marker = "python_version > \"3.6\"" files = [ {file = "executing-2.0.1-py2.py3-none-any.whl", hash = "sha256:eac49ca94516ccc753f9fb5ce82603156e590b27525a8bc32cce8ae302eb61bc"}, {file = "executing-2.0.1.tar.gz", hash = "sha256:35afe2ce3affba8ee97f2d69927fa823b08b472b7b994e36a52a964b93d16147"}, @@ -294,6 +327,8 @@ name = "fasteners" version = "0.19" requires_python = ">=3.6" summary = "A python package that provides useful locks" +groups = ["default"] +marker = "sys_platform != \"emscripten\"" files = [ {file = "fasteners-0.19-py3-none-any.whl", hash = "sha256:758819cb5d94cdedf4e836988b74de396ceacb8e2794d21f82d131fd9ee77237"}, {file = "fasteners-0.19.tar.gz", hash = "sha256:b4f37c3ac52d8a445af3a66bce57b33b5e90b97c696b7b984f530cf8f0ded09c"}, @@ -304,6 +339,7 @@ name = "filelock" version = "3.15.4" requires_python = ">=3.8" summary = "A platform independent file lock." +groups = ["dev"] files = [ {file = "filelock-3.15.4-py3-none-any.whl", hash = "sha256:6ca1fffae96225dab4c6eaf1c4f4f28cd2568d3ec2a44e15a08520504de468e7"}, {file = "filelock-3.15.4.tar.gz", hash = "sha256:2207938cbc1844345cb01a5a95524dae30f0ce089eba5b00378295a17e3e90cb"}, @@ -314,6 +350,7 @@ name = "frozenlist" version = "1.4.1" requires_python = ">=3.8" summary = "A list-like structure which implements collections.abc.MutableSequence" +groups = ["default"] files = [ {file = "frozenlist-1.4.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:f9aa1878d1083b276b0196f2dfbe00c9b7e752475ed3b682025ff20c1c1f51ac"}, {file = "frozenlist-1.4.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:29acab3f66f0f24674b7dc4736477bcd4bc3ad4b896f5f45379a67bce8b96868"}, @@ -366,12 +403,13 @@ files = [ [[package]] name = "fsspec" -version = "2024.2.0" +version = "2024.6.0" requires_python = ">=3.8" summary = "File-system specification" +groups = ["default"] files = [ - {file = "fsspec-2024.2.0-py3-none-any.whl", hash = "sha256:817f969556fa5916bc682e02ca2045f96ff7f586d45110fcb76022063ad2c7d8"}, - {file = "fsspec-2024.2.0.tar.gz", hash = "sha256:b6ad1a679f760dda52b1168c859d01b7b80648ea6f7f7c7f5a8a91dc3f3ecb84"}, + {file = "fsspec-2024.6.0-py3-none-any.whl", hash = "sha256:58d7122eb8a1a46f7f13453187bfea4972d66bf01618d37366521b1998034cee"}, + {file = "fsspec-2024.6.0.tar.gz", hash = "sha256:f579960a56e6d8038a9efc8f9c77279ec12e6299aa86b0769a7e9c46b94527c2"}, ] [[package]] @@ -379,6 +417,7 @@ name = "identify" version = "2.5.36" requires_python = ">=3.8" summary = "File identification library for Python" +groups = ["dev"] files = [ {file = "identify-2.5.36-py2.py3-none-any.whl", hash = "sha256:37d93f380f4de590500d9dba7db359d0d3da95ffe7f9de1753faa159e71e7dfa"}, {file = "identify-2.5.36.tar.gz", hash = "sha256:e5e00f54165f9047fbebeb4a560f9acfb8af4c88232be60a488e9b68d122745d"}, @@ -386,25 +425,28 @@ files = [ [[package]] name = "idna" -version = "3.6" +version = "3.7" requires_python = ">=3.5" summary = "Internationalized Domain Names in Applications (IDNA)" +groups = ["default"] files = [ - {file = "idna-3.6-py3-none-any.whl", hash = "sha256:c05567e9c24a6b9faaa835c4821bad0590fbb9d5779e7caa6e1cc4978e7eb24f"}, - {file = "idna-3.6.tar.gz", hash = "sha256:9ecdbbd083b06798ae1e86adcbfe8ab1479cf864e4ee30fe4e46a003d12491ca"}, + {file = "idna-3.7-py3-none-any.whl", hash = "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"}, + {file = "idna-3.7.tar.gz", hash = "sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc"}, ] [[package]] name = "importlib-metadata" -version = "7.0.1" +version = "7.2.1" requires_python = ">=3.8" summary = "Read metadata from Python packages" +groups = ["default"] +marker = "python_version < \"3.12\"" dependencies = [ "zipp>=0.5", ] files = [ - {file = "importlib_metadata-7.0.1-py3-none-any.whl", hash = "sha256:4805911c3a4ec7c3966410053e9ec6a1fecd629117df5adee56dfc9432a1081e"}, - {file = "importlib_metadata-7.0.1.tar.gz", hash = "sha256:f238736bb06590ae52ac1fab06a3a9ef1d8dce2b7a35b5ab329371d6c8f5d2cc"}, + {file = "importlib_metadata-7.2.1-py3-none-any.whl", hash = "sha256:ffef94b0b66046dd8ea2d619b701fe978d9264d38f3998bc4c27ec3b146a87c8"}, + {file = "importlib_metadata-7.2.1.tar.gz", hash = "sha256:509ecb2ab77071db5137c655e24ceb3eee66e7bbc6574165d0d114d9fc4bbe68"}, ] [[package]] @@ -412,6 +454,7 @@ name = "iniconfig" version = "2.0.0" requires_python = ">=3.7" summary = "brain-dead simple config-ini parsing" +groups = ["dev"] files = [ {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, @@ -422,6 +465,7 @@ name = "ipdb" version = "0.13.13" requires_python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" summary = "IPython-enabled pdb" +groups = ["dev"] dependencies = [ "decorator; python_version > \"3.6\" and python_version < \"3.11\"", "decorator; python_version >= \"3.11\"", @@ -436,9 +480,11 @@ files = [ [[package]] name = "ipython" -version = "8.22.1" +version = "8.25.0" requires_python = ">=3.10" summary = "IPython: Productive Interactive Computing" +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "colorama; sys_platform == \"win32\"", "decorator", @@ -450,16 +496,18 @@ dependencies = [ "pygments>=2.4.0", "stack-data", "traitlets>=5.13.0", + "typing-extensions>=4.6; python_version < \"3.12\"", ] files = [ - {file = "ipython-8.22.1-py3-none-any.whl", hash = "sha256:869335e8cded62ffb6fac8928e5287a05433d6462e3ebaac25f4216474dd6bc4"}, - {file = "ipython-8.22.1.tar.gz", hash = "sha256:39c6f9efc079fb19bfb0f17eee903978fe9a290b1b82d68196c641cecb76ea22"}, + {file = "ipython-8.25.0-py3-none-any.whl", hash = "sha256:53eee7ad44df903a06655871cbab66d156a051fd86f3ec6750470ac9604ac1ab"}, + {file = "ipython-8.25.0.tar.gz", hash = "sha256:c6ed726a140b6e725b911528f80439c534fac915246af3efc39440a6b0f9d716"}, ] [[package]] name = "isodate" version = "0.6.1" summary = "An ISO 8601 date/time/duration parser and formatter" +groups = ["default"] dependencies = [ "six", ] @@ -473,6 +521,8 @@ name = "jedi" version = "0.19.1" requires_python = ">=3.6" summary = "An autocompletion tool for Python that can be used for text editors." +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "parso<0.9.0,>=0.8.3", ] @@ -486,6 +536,7 @@ name = "locket" version = "1.0.0" requires_python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" summary = "File-based locks for Python on Linux and Windows" +groups = ["default"] files = [ {file = "locket-1.0.0-py2.py3-none-any.whl", hash = "sha256:b6c819a722f7b6bd955b80781788e4a66a55628b858d347536b7e81325a3a5e3"}, {file = "locket-1.0.0.tar.gz", hash = "sha256:5c0d4c052a8bbbf750e056a8e65ccd309086f4f0f18a2eac306a8dfa4112a632"}, @@ -496,6 +547,7 @@ name = "loguru" version = "0.7.2" requires_python = ">=3.5" summary = "Python logging made (stupidly) simple" +groups = ["default"] dependencies = [ "colorama>=0.3.4; sys_platform == \"win32\"", "win32-setctime>=1.0.0; sys_platform == \"win32\"", @@ -510,6 +562,7 @@ name = "markdown-it-py" version = "3.0.0" requires_python = ">=3.8" summary = "Python port of markdown-it. Markdown parsing, done right!" +groups = ["dev"] dependencies = [ "mdurl~=0.1", ] @@ -520,15 +573,17 @@ files = [ [[package]] name = "matplotlib-inline" -version = "0.1.6" -requires_python = ">=3.5" +version = "0.1.7" +requires_python = ">=3.8" summary = "Inline Matplotlib backend for Jupyter" +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "traitlets", ] files = [ - {file = "matplotlib-inline-0.1.6.tar.gz", hash = "sha256:f887e5f10ba98e8d2b150ddcf4702c1e5f8b3a20005eb0f74bfdbd360ee6f304"}, - {file = "matplotlib_inline-0.1.6-py3-none-any.whl", hash = "sha256:f1f41aab5328aa5aaea9b16d083b128102f8712542f819fe7e6a420ff581b311"}, + {file = "matplotlib_inline-0.1.7-py3-none-any.whl", hash = "sha256:df192d39a4ff8f21b1895d72e6a13f5fcc5099f00fa84384e0ea28c2cc0653ca"}, + {file = "matplotlib_inline-0.1.7.tar.gz", hash = "sha256:8423b23ec666be3d16e16b60bdd8ac4e86e840ebd1dd11a30b9f117f2fa0ab90"}, ] [[package]] @@ -536,6 +591,7 @@ name = "mdurl" version = "0.1.2" requires_python = ">=3.7" summary = "Markdown URL utilities" +groups = ["dev"] files = [ {file = "mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8"}, {file = "mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba"}, @@ -546,6 +602,7 @@ name = "multidict" version = "6.0.5" requires_python = ">=3.7" summary = "multidict implementation" +groups = ["default"] files = [ {file = "multidict-6.0.5-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:228b644ae063c10e7f324ab1ab6b548bdf6f8b47f3ec234fef1093bc2735e5f9"}, {file = "multidict-6.0.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:896ebdcf62683551312c30e20614305f53125750803b614e9e6ce74a96232604"}, @@ -601,6 +658,7 @@ name = "nodeenv" version = "1.9.1" requires_python = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" summary = "Node.js virtual environment builder" +groups = ["dev"] files = [ {file = "nodeenv-1.9.1-py2.py3-none-any.whl", hash = "sha256:ba11c9782d29c27c70ffbdda2d7415098754709be8a7056d79a737cd901155c9"}, {file = "nodeenv-1.9.1.tar.gz", hash = "sha256:6ec12890a2dab7946721edbfbcd91f3319c6ccc9aec47be7c7e6b7011ee6645f"}, @@ -611,6 +669,7 @@ name = "numcodecs" version = "0.12.1" requires_python = ">=3.8" summary = "A Python package providing buffer compression and transformation codecs for use in data storage and communication applications." +groups = ["default"] dependencies = [ "numpy>=1.7", ] @@ -632,116 +691,131 @@ files = [ [[package]] name = "numpy" -version = "1.26.4" +version = "2.0.0" requires_python = ">=3.9" summary = "Fundamental package for array computing in Python" -files = [ - {file = "numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0"}, - {file = "numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a"}, - {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4"}, - {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f"}, - {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a"}, - {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2"}, - {file = "numpy-1.26.4-cp310-cp310-win32.whl", hash = "sha256:bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07"}, - {file = "numpy-1.26.4-cp310-cp310-win_amd64.whl", hash = "sha256:b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5"}, - {file = "numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71"}, - {file = "numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef"}, - {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e"}, - {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5"}, - {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a"}, - {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a"}, - {file = "numpy-1.26.4-cp311-cp311-win32.whl", hash = "sha256:1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20"}, - {file = "numpy-1.26.4-cp311-cp311-win_amd64.whl", hash = "sha256:cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2"}, - {file = "numpy-1.26.4-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218"}, - {file = "numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b"}, - {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b"}, - {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed"}, - {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a"}, - {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0"}, - {file = "numpy-1.26.4-cp312-cp312-win32.whl", hash = "sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110"}, - {file = "numpy-1.26.4-cp312-cp312-win_amd64.whl", hash = "sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818"}, - {file = "numpy-1.26.4-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:afedb719a9dcfc7eaf2287b839d8198e06dcd4cb5d276a3df279231138e83d30"}, - {file = "numpy-1.26.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:95a7476c59002f2f6c590b9b7b998306fba6a5aa646b1e22ddfeaf8f78c3a29c"}, - {file = "numpy-1.26.4-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:7e50d0a0cc3189f9cb0aeb3a6a6af18c16f59f004b866cd2be1c14b36134a4a0"}, - {file = "numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010"}, +groups = ["default"] +files = [ + {file = "numpy-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:04494f6ec467ccb5369d1808570ae55f6ed9b5809d7f035059000a37b8d7e86f"}, + {file = "numpy-2.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2635dbd200c2d6faf2ef9a0d04f0ecc6b13b3cad54f7c67c61155138835515d2"}, + {file = "numpy-2.0.0-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:0a43f0974d501842866cc83471bdb0116ba0dffdbaac33ec05e6afed5b615238"}, + {file = "numpy-2.0.0-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:8d83bb187fb647643bd56e1ae43f273c7f4dbcdf94550d7938cfc32566756514"}, + {file = "numpy-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79e843d186c8fb1b102bef3e2bc35ef81160ffef3194646a7fdd6a73c6b97196"}, + {file = "numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1"}, + {file = "numpy-2.0.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:b4c76e3d4c56f145d41b7b6751255feefae92edbc9a61e1758a98204200f30fc"}, + {file = "numpy-2.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:acd3a644e4807e73b4e1867b769fbf1ce8c5d80e7caaef0d90dcdc640dfc9787"}, + {file = "numpy-2.0.0-cp310-cp310-win32.whl", hash = "sha256:cee6cc0584f71adefe2c908856ccc98702baf95ff80092e4ca46061538a2ba98"}, + {file = "numpy-2.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:ed08d2703b5972ec736451b818c2eb9da80d66c3e84aed1deeb0c345fefe461b"}, + {file = "numpy-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ad0c86f3455fbd0de6c31a3056eb822fc939f81b1618f10ff3406971893b62a5"}, + {file = "numpy-2.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e7f387600d424f91576af20518334df3d97bc76a300a755f9a8d6e4f5cadd289"}, + {file = "numpy-2.0.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:34f003cb88b1ba38cb9a9a4a3161c1604973d7f9d5552c38bc2f04f829536609"}, + {file = "numpy-2.0.0-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:b6f6a8f45d0313db07d6d1d37bd0b112f887e1369758a5419c0370ba915b3871"}, + {file = "numpy-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5f64641b42b2429f56ee08b4f427a4d2daf916ec59686061de751a55aafa22e4"}, + {file = "numpy-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581"}, + {file = "numpy-2.0.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:46e161722e0f619749d1cd892167039015b2c2817296104487cd03ed4a955995"}, + {file = "numpy-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0e50842b2295ba8414c8c1d9d957083d5dfe9e16828b37de883f51fc53c4016f"}, + {file = "numpy-2.0.0-cp311-cp311-win32.whl", hash = "sha256:2ce46fd0b8a0c947ae047d222f7136fc4d55538741373107574271bc00e20e8f"}, + {file = "numpy-2.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:fbd6acc766814ea6443628f4e6751d0da6593dae29c08c0b2606164db026970c"}, + {file = "numpy-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:354f373279768fa5a584bac997de6a6c9bc535c482592d7a813bb0c09be6c76f"}, + {file = "numpy-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4d2f62e55a4cd9c58c1d9a1c9edaedcd857a73cb6fda875bf79093f9d9086f85"}, + {file = "numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:1e72728e7501a450288fc8e1f9ebc73d90cfd4671ebbd631f3e7857c39bd16f2"}, + {file = "numpy-2.0.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:84554fc53daa8f6abf8e8a66e076aff6ece62de68523d9f665f32d2fc50fd66e"}, + {file = "numpy-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c73aafd1afca80afecb22718f8700b40ac7cab927b8abab3c3e337d70e10e5a2"}, + {file = "numpy-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49d9f7d256fbc804391a7f72d4a617302b1afac1112fac19b6c6cec63fe7fe8a"}, + {file = "numpy-2.0.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:0ec84b9ba0654f3b962802edc91424331f423dcf5d5f926676e0150789cb3d95"}, + {file = "numpy-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:feff59f27338135776f6d4e2ec7aeeac5d5f7a08a83e80869121ef8164b74af9"}, + {file = "numpy-2.0.0-cp312-cp312-win32.whl", hash = "sha256:c5a59996dc61835133b56a32ebe4ef3740ea5bc19b3983ac60cc32be5a665d54"}, + {file = "numpy-2.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:a356364941fb0593bb899a1076b92dfa2029f6f5b8ba88a14fd0984aaf76d0df"}, + {file = "numpy-2.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:9416a5c2e92ace094e9f0082c5fd473502c91651fb896bc17690d6fc475128d6"}, + {file = "numpy-2.0.0-pp39-pypy39_pp73-macosx_14_0_x86_64.whl", hash = "sha256:17067d097ed036636fa79f6a869ac26df7db1ba22039d962422506640314933a"}, + {file = "numpy-2.0.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:38ecb5b0582cd125f67a629072fed6f83562d9dd04d7e03256c9829bdec027ad"}, + {file = "numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:cef04d068f5fb0518a77857953193b6bb94809a806bd0a14983a8f12ada060c9"}, + {file = "numpy-2.0.0.tar.gz", hash = "sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864"}, ] [[package]] name = "packaging" -version = "23.2" -requires_python = ">=3.7" +version = "24.1" +requires_python = ">=3.8" summary = "Core utilities for Python packages" +groups = ["default", "dev"] files = [ - {file = "packaging-23.2-py3-none-any.whl", hash = "sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7"}, - {file = "packaging-23.2.tar.gz", hash = "sha256:048fb0e9405036518eaaf48a55953c750c11e1a1b68e0dd1a9d62ed0c092cfc5"}, + {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, + {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, ] [[package]] name = "pandas" -version = "2.2.1" +version = "2.2.2" requires_python = ">=3.9" summary = "Powerful data structures for data analysis, time series, and statistics" +groups = ["default"] dependencies = [ - "numpy<2,>=1.22.4; python_version < \"3.11\"", - "numpy<2,>=1.23.2; python_version == \"3.11\"", - "numpy<2,>=1.26.0; python_version >= \"3.12\"", + "numpy>=1.22.4; python_version < \"3.11\"", + "numpy>=1.23.2; python_version == \"3.11\"", + "numpy>=1.26.0; python_version >= \"3.12\"", "python-dateutil>=2.8.2", "pytz>=2020.1", "tzdata>=2022.7", ] files = [ - {file = "pandas-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:8df8612be9cd1c7797c93e1c5df861b2ddda0b48b08f2c3eaa0702cf88fb5f88"}, - {file = "pandas-2.2.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0f573ab277252ed9aaf38240f3b54cfc90fff8e5cab70411ee1d03f5d51f3944"}, - {file = "pandas-2.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f02a3a6c83df4026e55b63c1f06476c9aa3ed6af3d89b4f04ea656ccdaaaa359"}, - {file = "pandas-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c38ce92cb22a4bea4e3929429aa1067a454dcc9c335799af93ba9be21b6beb51"}, - {file = "pandas-2.2.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:c2ce852e1cf2509a69e98358e8458775f89599566ac3775e70419b98615f4b06"}, - {file = "pandas-2.2.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:53680dc9b2519cbf609c62db3ed7c0b499077c7fefda564e330286e619ff0dd9"}, - {file = "pandas-2.2.1-cp310-cp310-win_amd64.whl", hash = "sha256:94e714a1cca63e4f5939cdce5f29ba8d415d85166be3441165edd427dc9f6bc0"}, - {file = "pandas-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f821213d48f4ab353d20ebc24e4faf94ba40d76680642fb7ce2ea31a3ad94f9b"}, - {file = "pandas-2.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c70e00c2d894cb230e5c15e4b1e1e6b2b478e09cf27cc593a11ef955b9ecc81a"}, - {file = "pandas-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e97fbb5387c69209f134893abc788a6486dbf2f9e511070ca05eed4b930b1b02"}, - {file = "pandas-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:101d0eb9c5361aa0146f500773395a03839a5e6ecde4d4b6ced88b7e5a1a6403"}, - {file = "pandas-2.2.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:7d2ed41c319c9fb4fd454fe25372028dfa417aacb9790f68171b2e3f06eae8cd"}, - {file = "pandas-2.2.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:af5d3c00557d657c8773ef9ee702c61dd13b9d7426794c9dfeb1dc4a0bf0ebc7"}, - {file = "pandas-2.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:06cf591dbaefb6da9de8472535b185cba556d0ce2e6ed28e21d919704fef1a9e"}, - {file = "pandas-2.2.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:88ecb5c01bb9ca927ebc4098136038519aa5d66b44671861ffab754cae75102c"}, - {file = "pandas-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:04f6ec3baec203c13e3f8b139fb0f9f86cd8c0b94603ae3ae8ce9a422e9f5bee"}, - {file = "pandas-2.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a935a90a76c44fe170d01e90a3594beef9e9a6220021acfb26053d01426f7dc2"}, - {file = "pandas-2.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c391f594aae2fd9f679d419e9a4d5ba4bce5bb13f6a989195656e7dc4b95c8f0"}, - {file = "pandas-2.2.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9d1265545f579edf3f8f0cb6f89f234f5e44ba725a34d86535b1a1d38decbccc"}, - {file = "pandas-2.2.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:11940e9e3056576ac3244baef2fedade891977bcc1cb7e5cc8f8cc7d603edc89"}, - {file = "pandas-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:4acf681325ee1c7f950d058b05a820441075b0dd9a2adf5c4835b9bc056bf4fb"}, - {file = "pandas-2.2.1.tar.gz", hash = "sha256:0ab90f87093c13f3e8fa45b48ba9f39181046e8f3317d3aadb2fffbb1b978572"}, + {file = "pandas-2.2.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:90c6fca2acf139569e74e8781709dccb6fe25940488755716d1d354d6bc58bce"}, + {file = "pandas-2.2.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c7adfc142dac335d8c1e0dcbd37eb8617eac386596eb9e1a1b77791cf2498238"}, + {file = "pandas-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4abfe0be0d7221be4f12552995e58723c7422c80a659da13ca382697de830c08"}, + {file = "pandas-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8635c16bf3d99040fdf3ca3db669a7250ddf49c55dc4aa8fe0ae0fa8d6dcc1f0"}, + {file = "pandas-2.2.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:40ae1dffb3967a52203105a077415a86044a2bea011b5f321c6aa64b379a3f51"}, + {file = "pandas-2.2.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:8e5a0b00e1e56a842f922e7fae8ae4077aee4af0acb5ae3622bd4b4c30aedf99"}, + {file = "pandas-2.2.2-cp310-cp310-win_amd64.whl", hash = "sha256:ddf818e4e6c7c6f4f7c8a12709696d193976b591cc7dc50588d3d1a6b5dc8772"}, + {file = "pandas-2.2.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:696039430f7a562b74fa45f540aca068ea85fa34c244d0deee539cb6d70aa288"}, + {file = "pandas-2.2.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8e90497254aacacbc4ea6ae5e7a8cd75629d6ad2b30025a4a8b09aa4faf55151"}, + {file = "pandas-2.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:58b84b91b0b9f4bafac2a0ac55002280c094dfc6402402332c0913a59654ab2b"}, + {file = "pandas-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6d2123dc9ad6a814bcdea0f099885276b31b24f7edf40f6cdbc0912672e22eee"}, + {file = "pandas-2.2.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:2925720037f06e89af896c70bca73459d7e6a4be96f9de79e2d440bd499fe0db"}, + {file = "pandas-2.2.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:0cace394b6ea70c01ca1595f839cf193df35d1575986e484ad35c4aeae7266c1"}, + {file = "pandas-2.2.2-cp311-cp311-win_amd64.whl", hash = "sha256:873d13d177501a28b2756375d59816c365e42ed8417b41665f346289adc68d24"}, + {file = "pandas-2.2.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:9dfde2a0ddef507a631dc9dc4af6a9489d5e2e740e226ad426a05cabfbd7c8ef"}, + {file = "pandas-2.2.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e9b79011ff7a0f4b1d6da6a61aa1aa604fb312d6647de5bad20013682d1429ce"}, + {file = "pandas-2.2.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1cb51fe389360f3b5a4d57dbd2848a5f033350336ca3b340d1c53a1fad33bcad"}, + {file = "pandas-2.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eee3a87076c0756de40b05c5e9a6069c035ba43e8dd71c379e68cab2c20f16ad"}, + {file = "pandas-2.2.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:3e374f59e440d4ab45ca2fffde54b81ac3834cf5ae2cdfa69c90bc03bde04d76"}, + {file = "pandas-2.2.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:43498c0bdb43d55cb162cdc8c06fac328ccb5d2eabe3cadeb3529ae6f0517c32"}, + {file = "pandas-2.2.2-cp312-cp312-win_amd64.whl", hash = "sha256:d187d355ecec3629624fccb01d104da7d7f391db0311145817525281e2804d23"}, + {file = "pandas-2.2.2.tar.gz", hash = "sha256:9e79019aba43cb4fda9e4d983f8e88ca0373adbb697ae9c6c43093218de28b54"}, ] [[package]] name = "parso" -version = "0.8.3" +version = "0.8.4" requires_python = ">=3.6" summary = "A Python Parser" +groups = ["dev"] +marker = "python_version > \"3.6\"" files = [ - {file = "parso-0.8.3-py2.py3-none-any.whl", hash = "sha256:c001d4636cd3aecdaf33cbb40aebb59b094be2a74c556778ef5576c175e19e75"}, - {file = "parso-0.8.3.tar.gz", hash = "sha256:8c07be290bb59f03588915921e29e8a50002acaf2cdc5fa0e0114f91709fafa0"}, + {file = "parso-0.8.4-py2.py3-none-any.whl", hash = "sha256:a418670a20291dacd2dddc80c377c5c3791378ee1e8d12bffc35420643d43f18"}, + {file = "parso-0.8.4.tar.gz", hash = "sha256:eb3a7b58240fb99099a345571deecc0f9540ea5f4dd2fe14c2a99d6b281ab92d"}, ] [[package]] name = "partd" -version = "1.4.1" -requires_python = ">=3.7" +version = "1.4.2" +requires_python = ">=3.9" summary = "Appendable key-value storage" +groups = ["default"] dependencies = [ "locket", "toolz", ] files = [ - {file = "partd-1.4.1-py3-none-any.whl", hash = "sha256:27e766663d36c161e2827aa3e28541c992f0b9527d3cca047e13fb3acdb989e6"}, - {file = "partd-1.4.1.tar.gz", hash = "sha256:56c25dd49e6fea5727e731203c466c6e092f308d8f0024e199d02f6aa2167f67"}, + {file = "partd-1.4.2-py3-none-any.whl", hash = "sha256:978e4ac767ec4ba5b86c6eaa52e5a2a3bc748a2ca839e8cc798f1cc6ce6efb0f"}, + {file = "partd-1.4.2.tar.gz", hash = "sha256:d022c33afbdc8405c226621b015e8067888173d85f7f5ecebb3cafed9a20f02c"}, ] [[package]] name = "pexpect" version = "4.9.0" summary = "Pexpect allows easy control of interactive console applications." +groups = ["dev"] +marker = "(sys_platform != \"win32\" and sys_platform != \"emscripten\") and python_version > \"3.6\"" dependencies = [ "ptyprocess>=0.5", ] @@ -755,6 +829,7 @@ name = "platformdirs" version = "4.2.2" requires_python = ">=3.8" summary = "A small Python package for determining appropriate platform-specific dirs, e.g. a `user data dir`." +groups = ["dev"] files = [ {file = "platformdirs-4.2.2-py3-none-any.whl", hash = "sha256:2d7a1657e36a80ea911db832a8a6ece5ee53d8de21edd5cc5879af6530b1bfee"}, {file = "platformdirs-4.2.2.tar.gz", hash = "sha256:38b7b51f512eed9e84a22788b4bce1de17c0adb134d6becb09836e37d8654cd3"}, @@ -762,12 +837,13 @@ files = [ [[package]] name = "pluggy" -version = "1.4.0" +version = "1.5.0" requires_python = ">=3.8" summary = "plugin and hook calling mechanisms for python" +groups = ["dev"] files = [ - {file = "pluggy-1.4.0-py3-none-any.whl", hash = "sha256:7db9f7b503d67d1c5b95f59773ebb58a8c1c288129a88665838012cfb07b8981"}, - {file = "pluggy-1.4.0.tar.gz", hash = "sha256:8c85c2876142a764e5b7548e7d9a0e0ddb46f5185161049a79b7e974454223be"}, + {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, + {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, ] [[package]] @@ -775,6 +851,7 @@ name = "pre-commit" version = "3.7.1" requires_python = ">=3.9" summary = "A framework for managing and maintaining multi-language pre-commit hooks." +groups = ["dev"] dependencies = [ "cfgv>=2.0.0", "identify>=1.0.0", @@ -789,21 +866,25 @@ files = [ [[package]] name = "prompt-toolkit" -version = "3.0.43" +version = "3.0.47" requires_python = ">=3.7.0" summary = "Library for building powerful interactive command lines in Python" +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "wcwidth", ] files = [ - {file = "prompt_toolkit-3.0.43-py3-none-any.whl", hash = "sha256:a11a29cb3bf0a28a387fe5122cdb649816a957cd9261dcedf8c9f1fef33eacf6"}, - {file = "prompt_toolkit-3.0.43.tar.gz", hash = "sha256:3527b7af26106cbc65a040bcc84839a3566ec1b051bb0bfe953631e704b0ff7d"}, + {file = "prompt_toolkit-3.0.47-py3-none-any.whl", hash = "sha256:0d7bfa67001d5e39d02c224b663abc33687405033a8c422d0d675a5a13361d10"}, + {file = "prompt_toolkit-3.0.47.tar.gz", hash = "sha256:1e1b29cb58080b1e69f207c893a1a7bf16d127a5c30c9d17a25a5d77792e5360"}, ] [[package]] name = "ptyprocess" version = "0.7.0" summary = "Run a subprocess in a pseudo terminal" +groups = ["dev"] +marker = "(sys_platform != \"win32\" and sys_platform != \"emscripten\") and python_version > \"3.6\"" files = [ {file = "ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35"}, {file = "ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220"}, @@ -813,6 +894,8 @@ files = [ name = "pure-eval" version = "0.2.2" summary = "Safely evaluate AST nodes without side effects" +groups = ["dev"] +marker = "python_version > \"3.6\"" files = [ {file = "pure_eval-0.2.2-py3-none-any.whl", hash = "sha256:01eaab343580944bc56080ebe0a674b39ec44a945e6d09ba7db3cb8cec289350"}, {file = "pure_eval-0.2.2.tar.gz", hash = "sha256:2b45320af6dfaa1750f543d714b6d1c520a1688dec6fd24d339063ce0aaa9ac3"}, @@ -820,49 +903,53 @@ files = [ [[package]] name = "pygments" -version = "2.17.2" -requires_python = ">=3.7" +version = "2.18.0" +requires_python = ">=3.8" summary = "Pygments is a syntax highlighting package written in Python." +groups = ["dev"] files = [ - {file = "pygments-2.17.2-py3-none-any.whl", hash = "sha256:b27c2826c47d0f3219f29554824c30c5e8945175d888647acd804ddd04af846c"}, - {file = "pygments-2.17.2.tar.gz", hash = "sha256:da46cec9fd2de5be3a8a784f434e4c4ab670b4ff54d605c4c2717e9d49c4c367"}, + {file = "pygments-2.18.0-py3-none-any.whl", hash = "sha256:b8e6aca0523f3ab76fee51799c488e38782ac06eafcf95e7ba832985c8e7b13a"}, + {file = "pygments-2.18.0.tar.gz", hash = "sha256:786ff802f32e91311bff3889f6e9a86e81505fe99f2735bb6d60ae0c5004f199"}, ] [[package]] name = "pytest" -version = "8.0.2" +version = "8.2.2" requires_python = ">=3.8" summary = "pytest: simple powerful testing with Python" +groups = ["dev"] dependencies = [ "colorama; sys_platform == \"win32\"", "exceptiongroup>=1.0.0rc8; python_version < \"3.11\"", "iniconfig", "packaging", - "pluggy<2.0,>=1.3.0", - "tomli>=1.0.0; python_version < \"3.11\"", + "pluggy<2.0,>=1.5", + "tomli>=1; python_version < \"3.11\"", ] files = [ - {file = "pytest-8.0.2-py3-none-any.whl", hash = "sha256:edfaaef32ce5172d5466b5127b42e0d6d35ebbe4453f0e3505d96afd93f6b096"}, - {file = "pytest-8.0.2.tar.gz", hash = "sha256:d4051d623a2e0b7e51960ba963193b09ce6daeb9759a451844a21e4ddedfc1bd"}, + {file = "pytest-8.2.2-py3-none-any.whl", hash = "sha256:c434598117762e2bd304e526244f67bf66bbd7b5d6cf22138be51ff661980343"}, + {file = "pytest-8.2.2.tar.gz", hash = "sha256:de4bb8104e201939ccdc688b27a89a7be2079b22e2bd2b07f806b6ba71117977"}, ] [[package]] name = "python-dateutil" -version = "2.8.2" +version = "2.9.0.post0" requires_python = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7" summary = "Extensions to the standard Python datetime module" +groups = ["default"] dependencies = [ "six>=1.5", ] files = [ - {file = "python-dateutil-2.8.2.tar.gz", hash = "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86"}, - {file = "python_dateutil-2.8.2-py2.py3-none-any.whl", hash = "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"}, + {file = "python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3"}, + {file = "python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427"}, ] [[package]] name = "pytz" version = "2024.1" summary = "World timezone definitions, modern and historical" +groups = ["default"] files = [ {file = "pytz-2024.1-py2.py3-none-any.whl", hash = "sha256:328171f4e3623139da4983451950b28e95ac706e13f3f2630a879749e7a8b319"}, {file = "pytz-2024.1.tar.gz", hash = "sha256:2a29735ea9c18baf14b448846bde5a48030ed267578472d8955cd0e7443a9812"}, @@ -873,6 +960,7 @@ name = "pyyaml" version = "6.0.1" requires_python = ">=3.6" summary = "YAML parser and emitter for Python" +groups = ["default", "dev"] files = [ {file = "PyYAML-6.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a"}, {file = "PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"}, @@ -902,9 +990,10 @@ files = [ [[package]] name = "requests" -version = "2.31.0" -requires_python = ">=3.7" +version = "2.32.3" +requires_python = ">=3.8" summary = "Python HTTP for Humans." +groups = ["default"] dependencies = [ "certifi>=2017.4.17", "charset-normalizer<4,>=2", @@ -912,8 +1001,8 @@ dependencies = [ "urllib3<3,>=1.21.1", ] files = [ - {file = "requests-2.31.0-py3-none-any.whl", hash = "sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f"}, - {file = "requests-2.31.0.tar.gz", hash = "sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1"}, + {file = "requests-2.32.3-py3-none-any.whl", hash = "sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"}, + {file = "requests-2.32.3.tar.gz", hash = "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760"}, ] [[package]] @@ -921,6 +1010,7 @@ name = "rich" version = "13.7.1" requires_python = ">=3.7.0" summary = "Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal" +groups = ["dev"] dependencies = [ "markdown-it-py>=2.2.0", "pygments<3.0.0,>=2.13.0", @@ -930,11 +1020,23 @@ files = [ {file = "rich-13.7.1.tar.gz", hash = "sha256:9be308cb1fe2f1f57d67ce99e95af38a1e2bc71ad9813b0e247cf7ffbcc3a432"}, ] +[[package]] +name = "semver" +version = "3.0.2" +requires_python = ">=3.7" +summary = "Python helper for Semantic Versioning (https://semver.org)" +groups = ["default"] +files = [ + {file = "semver-3.0.2-py3-none-any.whl", hash = "sha256:b1ea4686fe70b981f85359eda33199d60c53964284e0cfb4977d243e37cf4bf4"}, + {file = "semver-3.0.2.tar.gz", hash = "sha256:6253adb39c70f6e51afed2fa7152bcd414c411286088fb4b9effb133885ab4cc"}, +] + [[package]] name = "six" version = "1.16.0" requires_python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*" summary = "Python 2 and 3 compatibility utilities" +groups = ["default", "dev"] files = [ {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"}, {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"}, @@ -944,6 +1046,8 @@ files = [ name = "stack-data" version = "0.6.3" summary = "Extract data from python stack frames and tracebacks for informative displays" +groups = ["dev"] +marker = "python_version > \"3.6\"" dependencies = [ "asttokens>=2.1.0", "executing>=1.2.0", @@ -959,6 +1063,8 @@ name = "tomli" version = "2.0.1" requires_python = ">=3.7" summary = "A lil' TOML parser" +groups = ["dev"] +marker = "python_version < \"3.11\"" files = [ {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"}, {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"}, @@ -969,6 +1075,7 @@ name = "toolz" version = "0.12.1" requires_python = ">=3.7" summary = "List processing tools and functional utilities" +groups = ["default"] files = [ {file = "toolz-0.12.1-py3-none-any.whl", hash = "sha256:d22731364c07d72eea0a0ad45bafb2c2937ab6fd38a3507bf55eae8744aa7d85"}, {file = "toolz-0.12.1.tar.gz", hash = "sha256:ecca342664893f177a13dac0e6b41cbd8ac25a358e5f215316d43e2100224f4d"}, @@ -976,12 +1083,26 @@ files = [ [[package]] name = "traitlets" -version = "5.14.1" +version = "5.14.3" requires_python = ">=3.8" summary = "Traitlets Python configuration system" +groups = ["dev"] +marker = "python_version > \"3.6\"" +files = [ + {file = "traitlets-5.14.3-py3-none-any.whl", hash = "sha256:b74e89e397b1ed28cc831db7aea759ba6640cb3de13090ca145426688ff1ac4f"}, + {file = "traitlets-5.14.3.tar.gz", hash = "sha256:9ed0579d3502c94b4b3732ac120375cda96f923114522847de4b3bb98b96b6b7"}, +] + +[[package]] +name = "typing-extensions" +version = "4.12.2" +requires_python = ">=3.8" +summary = "Backported and Experimental Type Hints for Python 3.8+" +groups = ["dev"] +marker = "python_version < \"3.12\" and python_version > \"3.6\"" files = [ - {file = "traitlets-5.14.1-py3-none-any.whl", hash = "sha256:2e5a030e6eff91737c643231bfcf04a65b0132078dad75e4936700b213652e74"}, - {file = "traitlets-5.14.1.tar.gz", hash = "sha256:8585105b371a04b8316a43d5ce29c098575c2e477850b62b848b964f1444527e"}, + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, ] [[package]] @@ -989,6 +1110,7 @@ name = "tzdata" version = "2024.1" requires_python = ">=2" summary = "Provider of IANA time zone data" +groups = ["default"] files = [ {file = "tzdata-2024.1-py2.py3-none-any.whl", hash = "sha256:9068bc196136463f5245e51efda838afa15aaeca9903f49050dfa2679db4d252"}, {file = "tzdata-2024.1.tar.gz", hash = "sha256:2674120f8d891909751c38abcdfd386ac0a5a1127954fbc332af6b5ceae07efd"}, @@ -996,12 +1118,13 @@ files = [ [[package]] name = "urllib3" -version = "2.2.1" +version = "2.2.2" requires_python = ">=3.8" summary = "HTTP library with thread-safe connection pooling, file post, and more." +groups = ["default"] files = [ - {file = "urllib3-2.2.1-py3-none-any.whl", hash = "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d"}, - {file = "urllib3-2.2.1.tar.gz", hash = "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19"}, + {file = "urllib3-2.2.2-py3-none-any.whl", hash = "sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472"}, + {file = "urllib3-2.2.2.tar.gz", hash = "sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168"}, ] [[package]] @@ -1009,6 +1132,7 @@ name = "virtualenv" version = "20.26.3" requires_python = ">=3.7" summary = "Virtual Python Environment builder" +groups = ["dev"] dependencies = [ "distlib<1,>=0.3.7", "filelock<4,>=3.12.2", @@ -1023,6 +1147,8 @@ files = [ name = "wcwidth" version = "0.2.13" summary = "Measures the displayed width of unicode strings in a terminal" +groups = ["dev"] +marker = "python_version > \"3.6\"" files = [ {file = "wcwidth-0.2.13-py2.py3-none-any.whl", hash = "sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859"}, {file = "wcwidth-0.2.13.tar.gz", hash = "sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5"}, @@ -1033,6 +1159,8 @@ name = "win32-setctime" version = "1.1.0" requires_python = ">=3.5" summary = "A small Python utility to set file creation time on Windows" +groups = ["default"] +marker = "sys_platform == \"win32\"" files = [ {file = "win32_setctime-1.1.0-py3-none-any.whl", hash = "sha256:231db239e959c2fe7eb1d7dc129f11172354f98361c4fa2d6d2d7e278baa8aad"}, {file = "win32_setctime-1.1.0.tar.gz", hash = "sha256:15cf5750465118d6929ae4de4eb46e8edae9a5634350c01ba582df868e932cb2"}, @@ -1040,17 +1168,18 @@ files = [ [[package]] name = "xarray" -version = "2024.2.0" +version = "2024.6.0" requires_python = ">=3.9" summary = "N-D labeled arrays and datasets in Python" +groups = ["default"] dependencies = [ "numpy>=1.23", - "packaging>=22", - "pandas>=1.5", + "packaging>=23.1", + "pandas>=2.0", ] files = [ - {file = "xarray-2024.2.0-py3-none-any.whl", hash = "sha256:a31a9b37e39bd5aeb098070a75d6dd4d59019eb339d735b86108b9e0cb391f94"}, - {file = "xarray-2024.2.0.tar.gz", hash = "sha256:a105f02791082c888ebe2622090beaff2e7b68571488d62fe6afdab35b4b717f"}, + {file = "xarray-2024.6.0-py3-none-any.whl", hash = "sha256:721a7394e8ec3d592b2d8ebe21eed074ac077dc1bb1bd777ce00e41700b4866c"}, + {file = "xarray-2024.6.0.tar.gz", hash = "sha256:0b91e0bc4dc0296947947640fe31ec6e867ce258d2f7cbc10bedf4a6d68340c7"}, ] [[package]] @@ -1058,6 +1187,7 @@ name = "yarl" version = "1.9.4" requires_python = ">=3.7" summary = "Yet another URL library" +groups = ["default"] dependencies = [ "idna>=2.0", "multidict>=4.0", @@ -1114,26 +1244,29 @@ files = [ [[package]] name = "zarr" -version = "2.17.0" +version = "2.18.2" requires_python = ">=3.9" summary = "An implementation of chunked, compressed, N-dimensional arrays for Python" +groups = ["default"] dependencies = [ "asciitree", "fasteners; sys_platform != \"emscripten\"", "numcodecs>=0.10.0", - "numpy>=1.21.1", + "numpy>=1.23", ] files = [ - {file = "zarr-2.17.0-py3-none-any.whl", hash = "sha256:d287cb61019c4a0a0f386f76eeaa7f0b1160b1cb90cf96173a4b6cbc135df6e1"}, - {file = "zarr-2.17.0.tar.gz", hash = "sha256:6390a2b8af31babaab4c963efc45bf1da7f9500c9aafac193f84cf019a7c66b0"}, + {file = "zarr-2.18.2-py3-none-any.whl", hash = "sha256:a638754902f97efa99b406083fdc807a0e2ccf12a949117389d2a4ba9b05df38"}, + {file = "zarr-2.18.2.tar.gz", hash = "sha256:9bb393b8a0a38fb121dbb913b047d75db28de9890f6d644a217a73cf4ae74f47"}, ] [[package]] name = "zipp" -version = "3.17.0" +version = "3.19.2" requires_python = ">=3.8" summary = "Backport of pathlib-compatible object wrapper for zip files" +groups = ["default"] +marker = "python_version < \"3.12\"" files = [ - {file = "zipp-3.17.0-py3-none-any.whl", hash = "sha256:0e923e726174922dce09c53c59ad483ff7bbb8e572e00c7f7c46b88556409f31"}, - {file = "zipp-3.17.0.tar.gz", hash = "sha256:84e64a1c28cf7e91ed2078bb8cc8c259cb19b76942096c8d7b84947690cabaf0"}, + {file = "zipp-3.19.2-py3-none-any.whl", hash = "sha256:f091755f667055f2d02b32c53771a7a6c8b47e1fdbc4b72a8b9072b3eef8015c"}, + {file = "zipp-3.19.2.tar.gz", hash = "sha256:bf1dcf6450f873a13e952a29504887c89e6de7506209e5b1bcc3460135d4de19"}, ] diff --git a/pyproject.toml b/pyproject.toml index 5402637..8a078ec 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,6 +14,8 @@ dependencies = [ "isodate>=0.6.1", "requests>=2.31.0", "aiohttp>=3.9.3", + "dataclass-wizard>=0.22.3", + "semver>=3.0.2", ] requires-python = ">=3.10" readme = "README.md" diff --git a/tests/data.py b/tests/data.py index d08c4ae..c2517ff 100644 --- a/tests/data.py +++ b/tests/data.py @@ -11,6 +11,8 @@ DT_ANALYSIS = isodate.parse_duration("PT6H") DT_FORECAST = isodate.parse_duration("PT1H") T_START = isodate.parse_datetime("2000-01-01T00:00") +T_END_ANALYSIS = T_START + (NT_ANALYSIS - 1) * DT_ANALYSIS +T_END_FORECAST = T_START + (NT_FORECAST - 1) * DT_FORECAST DEFAULT_FORECAST_VARS = ["u", "v", "t", "precip"] DEFAULT_ATMOSPHERIC_ANALYSIS_VARS = ["u", "v", "t"] DEFAULT_SURFACE_ANALYSIS_VARS = [ diff --git a/tests/test_config.py b/tests/test_config.py index 6609882..9d340b8 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -1,18 +1,45 @@ import pytest -import yaml +from dataclass_wizard.errors import MissingFields, UnknownJSONKey -from mllam_data_prep.config import ( - ConfigDict, - InvalidConfigVariableException, - MissingConfigVariableException, -) +import mllam_data_prep as mdp -EXAMPLE_CONFIG_YAML = """ +INVALID_EXTRA_FIELDS_CONFIG_YAML = """ +schema_version: v0.1.0 +dataset_version: v0.1.0 + +architecture: + input_variables: + static: [grid_index, feature] + input_range: + time: + start: 1990-09-03T00:00 + end: 1990-09-04T00:00 + step: PT3H + +inputs: {} +foobar: 42 +""" + +MISSING_FIELDS_CONFIG_YAML = """ +schema_version: v0.1.0 +dataset_version: v0.1.0 +""" + + +def test_get_config_issues(): + """Test that the Config class raises the correct exceptions when the YAML file is invalid.""" + with pytest.raises(UnknownJSONKey): + mdp.Config.from_yaml(INVALID_EXTRA_FIELDS_CONFIG_YAML) + + with pytest.raises(MissingFields): + mdp.Config.from_yaml(MISSING_FIELDS_CONFIG_YAML) + + +VALID_EXAMPLE_CONFIG_YAML = """ schema_version: v0.1.0 dataset_version: v0.1.0 architecture: - sampling_dim: time input_variables: static: [grid_index, feature] state: [time, grid_index, state_feature] @@ -30,18 +57,20 @@ variables: u: altitude: - sel: [100, ] + values: [100, ] units: m v: altitude: - sel: [100, ] + values: [100, ] units: m dim_mapping: - time: time + time: + method: rename + dim: time state_feature: method: stack_variables_by_var_name dims: [altitude] - name: f"{var_name}{altitude}m" + name_format: f"{var_name}{altitude}m" grid_index: method: flatten dims: [x, y] @@ -53,37 +82,25 @@ variables: - pres_seasurface dim_mapping: - time: time + time: + method: rename + dim: time grid_index: method: flatten dims: [x, y] forcing_feature: method: stack_variables_by_var_name - name: f"{var_name}" + name_format: f"{var_name}" target_architecture_variable: forcing """ -def test_get_config(): - config = ConfigDict(dict(schema_version="v0.1.0", foobar=42)) - - assert config["schema_version"] is not None - # raise `InvalidConfig` because `dataset_version` is in the spec, but isn't defined in the config - with pytest.raises(MissingConfigVariableException): - config["dataset_version"] - - # raise `InvalidConfig` because although `foobar` is defined in the config, it shouldn't be as it isn't part of the spec - with pytest.raises(InvalidConfigVariableException): - config["foobar"] - - def test_get_config_nested(): - config = ConfigDict(yaml.load(EXAMPLE_CONFIG_YAML, Loader=yaml.FullLoader)) - - for dataset_name, input_config in config["inputs"].items(): - assert input_config["path"] is not None - assert input_config["variables"] is not None - assert input_config["target_architecture_variable"] is not None - with pytest.raises(KeyError): - # `name` is given by the key, so isn't expected to be its own field - input_config["name"] + config = mdp.Config.from_yaml(VALID_EXAMPLE_CONFIG_YAML) + + for dataset_name, input_config in config.inputs.items(): + assert input_config.path is not None + assert input_config.variables is not None + assert input_config.target_architecture_variable is not None + with pytest.raises(AttributeError): + input_config.foobarfield diff --git a/tests/test_from_config.py b/tests/test_from_config.py index 65485a8..ed6492a 100644 --- a/tests/test_from_config.py +++ b/tests/test_from_config.py @@ -7,7 +7,6 @@ import mllam_data_prep as mdp import tests.data as testdata -from mllam_data_prep.config import InvalidConfigException def test_gen_data(): @@ -24,26 +23,34 @@ def test_merging_static_and_surface_analysis(): ) config = dict( - schema_version="v0.1.0", + schema_version="v0.2.0", dataset_version="v0.1.0", architecture=dict( - sampling_dim="time", input_variables=dict( static=["grid_index", "static_feature"], state=["time", "grid_index", "state_feature"], forcing=["time", "grid_index", "forcing_feature"], ), + input_coord_ranges=dict( + time=dict( + start=testdata.T_START.isoformat(), + end=testdata.T_END_ANALYSIS.isoformat(), + step=isodate.duration_isoformat(testdata.DT_ANALYSIS), + ) + ), ), inputs=dict( danra_surface=dict( - name="danra_surface", path=datasets["surface_analysis"], dims=["analysis_time", "x", "y"], variables=testdata.DEFAULT_SURFACE_ANALYSIS_VARS, dim_mapping=dict( - time="analysis_time", + time=dict( + method="rename", + dim="analysis_time", + ), grid_index=dict( - method="flatten", + method="stack", dims=["x", "y"], ), forcing_feature=dict( @@ -54,13 +61,12 @@ def test_merging_static_and_surface_analysis(): target_architecture_variable="forcing", ), danra_static=dict( - name="danra_static", path=datasets["static"], dims=["x", "y"], variables=testdata.DEFAULT_STATIC_VARS, dim_mapping=dict( grid_index=dict( - method="flatten", + method="stack", dims=["x", "y"], ), static_feature=dict( @@ -110,16 +116,15 @@ def test_time_selection(source_data_contains_time_range, time_stepsize): t_end_config = t_end_dataset + testdata.DT_ANALYSIS config = dict( - schema_version="v0.1.0", + schema_version="v0.2.0", dataset_version="v0.1.0", architecture=dict( - sampling_dim="time", input_variables=dict( static=["grid_index", "feature"], state=["time", "grid_index", "feature"], forcing=["time", "grid_index", "feature"], ), - input_range=dict( + input_coord_ranges=dict( time=dict( start=t_start_config.isoformat(), end=t_end_config.isoformat(), @@ -133,9 +138,12 @@ def test_time_selection(source_data_contains_time_range, time_stepsize): dims=["analysis_time", "x", "y"], variables=testdata.DEFAULT_SURFACE_ANALYSIS_VARS, dim_mapping=dict( - time="analysis_time", + time=dict( + method="rename", + dim="analysis_time", + ), grid_index=dict( - method="flatten", + method="stack", dims=["x", "y"], ), feature=dict( @@ -184,10 +192,9 @@ def test_feature_collision(use_common_feature_var_name): state_feature_var_name = "state_feature" config = dict( - schema_version="v0.1.0", + schema_version="v0.2.0", dataset_version="v0.1.0", architecture=dict( - sampling_dim="time", input_variables=dict( static=["grid_index", static_feature_var_name], state=["time", "grid_index", state_feature_var_name], @@ -195,14 +202,16 @@ def test_feature_collision(use_common_feature_var_name): ), inputs=dict( danra_surface=dict( - name="danra_surface", path=datasets["surface_analysis"], dims=["analysis_time", "x", "y"], variables=testdata.DEFAULT_SURFACE_ANALYSIS_VARS, dim_mapping={ - "time": "analysis_time", + "time": dict( + method="rename", + dim="analysis_time", + ), "grid_index": dict( - method="flatten", + method="stack", dims=["x", "y"], ), state_feature_var_name: dict( @@ -213,18 +222,16 @@ def test_feature_collision(use_common_feature_var_name): target_architecture_variable="state", ), danra_static=dict( - name="danra_static", path=datasets["static"], dims=["x", "y"], variables=testdata.DEFAULT_STATIC_VARS, dim_mapping={ "grid_index": dict( dims=["x", "y"], - method="flatten", + method="stack", ), static_feature_var_name: dict( method="stack_variables_by_var_name", - stack_variables_by_var_name=True, name_format="{var_name}", ), }, @@ -240,7 +247,7 @@ def test_feature_collision(use_common_feature_var_name): yaml.dump(config, f) if use_common_feature_var_name: - with pytest.raises(InvalidConfigException): + with pytest.raises(mdp.InvalidConfigException): mdp.create_dataset_zarr(fp_config=fp_config) else: mdp.create_dataset_zarr(fp_config=fp_config) @@ -248,4 +255,5 @@ def test_feature_collision(use_common_feature_var_name): def test_danra_example(): fp_config = Path(__file__).parent.parent / "example.danra.yaml" - mdp.create_dataset_zarr(fp_config=fp_config) + with tempfile.TemporaryDirectory(suffix=".zarr") as tmpdir: + mdp.create_dataset_zarr(fp_config=fp_config, fp_zarr=tmpdir) diff --git a/tests/test_stacking.py b/tests/test_stacking.py index bee4473..d8fa859 100644 --- a/tests/test_stacking.py +++ b/tests/test_stacking.py @@ -1,6 +1,7 @@ import numpy as np import xarray as xr +from mllam_data_prep.config import DimMapping from mllam_data_prep.ops import mapping as mdp_mapping from mllam_data_prep.ops import stacking as mdp_stacking @@ -67,11 +68,11 @@ def test_stack_xy_coords(): coords={"level": np.arange(nz)}, ) dim_mapping = dict( - grid_index=dict( - method="flatten", + grid_index=DimMapping( + method="stack", dims=["x", "y"], ), - feature=dict( + feature=DimMapping( method="stack_variables_by_var_name", dims=["level"], name_format="{level}_{var_name}",