Manifest arrays use arrayv3metadata (#482)

* Manifest arrays use arrayv3metadata (#429)

* Added zarray_to_v3metadata and test

* Working on manifest array tests

* Fix test_manifests/test_array#TestConcat tests

* Passing TestStack tests and add fixture

* All test_manifests/test_array tests passing

* Compressors should be list

* Passing dmrpp tests

* Passing test_hdf.py tests

* Start to work on kerchunk tests

* Add method to convert array v3 metadata to v2 metadata for kerchunk (not happy about this)

* Fix fixtures and mark xfail netcdf3

* Test for convert_v3_to_v2_metadata

* Deduplicate fixture for array v3 metadata

* Parse filters and compressors from v3 metdata for v2 metadata

* Rewrite extract_codecs

* Refactor convert_to_codec_pipeline

* Fix hdf integration tests

* Test for convert_to_codec_pipeline

* Refactor get_codecs and its tests

* Fix most integration tests and writer tests

* Fix xarray tests

* Working on integration tests

* Add expected type

* Mark datetime tests xfail

* Upgrade xarray for tests

* xfail some unsupported zarr-python 3 data types

* Require zarr

* Remove zarr dep

* import zarr, explicit dependency

Co-authored-by: Tom Nicholas <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add zarr as a dependency

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Min numcodecs version

* numcodecs>=0.15.1 in environment and upstream.yml conda env files

* Working on mypy errors

* Fix mypy errors and tests

* Remove ZArray class

* Just return metadata's shape

* Create update metadata function

* Fix typing for update_metadata

* Check for regular chunk grid in manifest instantiation

* Remove obsolete codecs code

* Fix chunks function and add docstring

* Remove custom zattrs type

* Move some imports and make update_metadata a private method

* Remove zarr.py

* Add zarr to other ci env files

* Fixture array_v3_metadata uses array_v3_metadata_dict

* No need for union type for CodecPipeline

* Use type alias

* Add comment

* Update virtualizarr/manifests/array_api.py

Co-authored-by: Tom Nicholas <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revised copy_and_replace_metadata to be in utils and called correctly

* Update virtualizarr/translators/kerchunk.py

Co-authored-by: Tom Nicholas <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor create v3 array metadata

* Rename to create_v3_array_metadata

* Fix some codecs fixtures

* Use global vars and simple fixture for creating codec pipelines

* Remove redundant create_codec_pipeline fixture

* Fix docstring

* Use create_v3_array_metadata in from_kerchunk_refs

* Add links to zarr-python 3.0 issues for big endian, datetime and timedelta data types

* Reorganize conftest

* Remove obsolete comment

* Rename function numcodec_config_to_configurable

* Fix parameters in docstring for convert_to_codec_pipeline

* Revert change to pytest mark skipif for astropy

* Remove commented arguments

* Add classes to test_codecs and make zarr_array a fixture

* Add tests for extract_codecs

* Add test for get_codec_config

* Remove obsolete comment

* Add test for copy_and_replace_metadata

* Add release notes

* Attempt to fix rst links

* Move convert_v3_to_v2_metadata to utils

---------

Co-authored-by: Tom Nicholas <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: 3 people

ci/environment.yml

@@ -37,5 +37,6 @@ dependencies:
   # for opening FITS files
   - astropy
   - pip
+  - zarr>=3.0.2
   - pip:
       - imagecodecs-numcodecs==2024.6.1

ci/min-deps.yml

@@ -6,9 +6,9 @@ dependencies:
   - h5py
   - hdf5
   - netcdf4
-  - xarray>=2024.10.0
+  - xarray>=2025.1.1
   - numpy>=2.0.0
-  - numcodecs
+  - numcodecs>=0.15.1
   - packaging
   - ujson
   - universal_pathlib
@@ -23,3 +23,4 @@ dependencies:
   - pytest
   - pooch
   - fsspec
+  - zarr>=3.0.2

ci/upstream.yml

@@ -13,7 +13,7 @@ dependencies:
   - ujson
   - universal_pathlib
   - hdf5plugin
-  - numcodecs
+  - numcodecs>=0.15.1
   - imagecodecs>=2024.6.1
   # Testing
   - codecov[toml]
@@ -28,6 +28,7 @@ dependencies:
   - pooch
   - fsspec
   - pip
+  - zarr>=3.0.2
   - icechunk>=0.2.4
   - pip:
       - git+https://github.com/fsspec/kerchunk.git@main