Zarr group JSON file issues

[markgoddard]

PyActiveStorage writes out a Zarr group JSON metadata file when processing a variable in a dataset. The filename is <netcdf file basename>_<variable name>.json. If using local storage it is written to the same directory as the netCDF file. If using S3 storage it is written to the current directory. If a file of the same name exists, it is not updated. The files are never removed.

This leads to various issues when reusing netCDF filenames for different datasets (e.g. during testing), since the Zarr group metadata may be describing a previous incarnation of the dataset.

It also leads to an undesirable build up of JSON files.

Zarr dataset metadata is cached on an Active instance, so there isn't much to be gained from leaving the JSON files around to be reused.

I propose we use a temporary file to write the Zarr group metadata, and immediately remove it once it has been used.

[markgoddard]

To add some specifics, this lead to a difficult to diagnose issue when trying to implement #120. Because the same file name was used (test_compression.nc) for multiple test cases (with and without byte shuffled data), the Zarr metadata file would not be updated. This lead to incorrect chunk sizes and offsets, and ultimately decompression failure.

It's not clear why this did not also affect the test when using local storage.

[markgoddard]

It's not clear why this did not also affect the test when using local storage.

Worked it out (and updated the description) - for local files we generate the JSON file in the same directory as the netCDF file, which in the test case is in a temporary directory. This is not persistent, so does not suffer from the issue.