Variable names converted to lowercase when using to_zarr() on NetCDF files

[bmcandr]

After converting a multi-file NetCDF dataset to a consolidated Zarr store using xarray's to_zarr() function, I noticed that the variable names were converted from mixed-case to lowercase (e.g., SoilMoist_tavg became soilmoist_tavg).

open_mfdataset():

open_zarr():

The sample variable lists above appear different because the change in case results in a change in alphabetic sorting. All the variable fields from the NetCDF are present, however.

Some notes:

• I'm using xarray version 0.17.0
• The NetCDFs were hosted in S3 and the Zarr store was written to a new location on S3
• I did not encounter this when converting similar NetCDFs produced by the same model to Zarr on my local machine

Let me know if there is any additional information I can provide.

[andersy005]

@bmcandr, which version of zarr are you using? I'm asking because I am unable to reproduce the issue on my end:

In [6]: ds
Out[6]: 
<xarray.Dataset>
Dimensions:         (time: 36, x: 275, y: 205)
Coordinates:
  * time            (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00
    xc              (y, x) float64 ...
    yc              (y, x) float64 ...
Dimensions without coordinates: x, y
Data variables:
    SoilMoist_tavg  (time, y, x) float64 ...
Attributes:
    title:                     /workspace/jhamman/processed/R1002RBRxaaa01a/l...
    institution:               U.W.
    source:                    RACM R1002RBRxaaa01a
    output_frequency:          daily
    output_mode:               averaged
    convention:                CF-1.4
    references:                Based on the initial model of Liang et al., 19...
    comment:                   Output from the Variable Infiltration Capacity...
    nco_openmp_thread_number:  1
    NCO:                       "4.6.0"
    history:                   Tue Dec 27 14:15:22 2016: ncatted -a dimension...

In [7]: ds.to_zarr('test.zarr', consolidated=True, mode='w')
Out[7]: <xarray.backends.zarr.ZarrStore at 0x2b93487d8e80>

In [8]: xr.open_zarr('test.zarr', consolidated=True)
Out[8]: 
<xarray.Dataset>
Dimensions:         (time: 36, x: 275, y: 205)
Coordinates:
  * time            (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00
    xc              (y, x) float64 dask.array<chunksize=(103, 275), meta=np.ndarray>
    yc              (y, x) float64 dask.array<chunksize=(103, 275), meta=np.ndarray>
Dimensions without coordinates: x, y
Data variables:
    SoilMoist_tavg  (time, y, x) float64 dask.array<chunksize=(9, 52, 138), meta=np.ndarray>
Attributes:
    NCO:                       "4.6.0"
    comment:                   Output from the Variable Infiltration Capacity...
    convention:                CF-1.4
    history:                   Tue Dec 27 14:15:22 2016: ncatted -a dimension...
    institution:               U.W.
    nco_openmp_thread_number:  1
    output_frequency:          daily
    output_mode:               averaged
    references:                Based on the initial model of Liang et al., 19...
    source:                    RACM R1002RBRxaaa01a
    title:                     /workspace/jhamman/processed/R1002RBRxaaa01a/l...

In [9]: import zarr

In [10]: zarr.__version__
Out[10]: '2.8.1'

In [11]: xr.__version__
Out[11]: '0.17.0'

[bmcandr]

Will do.

[raybellwaves]

there is already an issue #5028

[bmcandr]

Thanks @raybellwaves.

[bmcandr]

I can confirm that the workaround described in #5028 does work, but it must be applied to both the to_zarr() and open_zarr() calls. For example:

test_path = 's3://bucket-name/TEST.zarr'
test_store = zarr.storage.FSStore(test_path, normalize_keys=False)

# write the dataset to zarr store
ds.to_zarr(store=test_store, consolidated=True)

# open the zarr store
xr.open_zarr(store=test_store, consolidated=True)

This writes and reads the store without normalizing the variable names.

Calling xr.open_zarr(test_path) on a store created as above runs without issue, BUT the data values are all nans. This is why it appears the workaround must be applied to both to_zarr() and open_zarr() calls.

[rabernat]

that is really weird and definitely a bug

[rabernat]

Perhaps related to s3fs normalizing path names somehow? cc @martindurant

[martindurant]

s3fs certainly allows capitals (and other special characters) in filenames.

[raybellwaves]

This should be fixed now (#5028 (comment))