diff --git a/docs/src/index.md b/docs/src/index.md index b4d6e2f..96b5873 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -1,15 +1,57 @@ ## ZarrDatasets + +See the [documentation of JuliaGeo/CommonDataModel.jl](https://juliageo.org/CommonDataModel.jl/stable/) for the full documentation of the API. As a quick reference, here is an example how to create and read a Zarr file store as a quick reference. + +### Create a Zarr file store + +The following example create a Zarr file store in the directory `"/tmp/test-zarr"`: + +```julia +using ZarrDatasets + +# sample data +data = [i+j for i = 1:3, j = 1:5] + +directoryname = "/tmp/test-zarr4" +mkdir(directoryname) + +ds = ZarrDataset(directoryname,"c") +defDim(ds,"lon",size(data,1)) +defDim(ds,"lat",size(data,2)) +zv = defVar(ds,"varname",Int64,("lon","lat")) +zv[:,:] = data +zv.attrib["units"] = "m" +close(ds) +``` + +### Loading a Zarr file store + +The data and units can be loaded by indexing the data set structure `ds`. + +```julia +using ZarrDatasets +directoryname = "/tmp/test-zarr4" +ds = ZarrDataset(directoryname) +data = ds["varname"][:,:] +data_units = ds["varname"].attrib["units"] +``` + + + ```@autodocs Modules = [ZarrDatasets] ``` + + + ### Differences between Zarr and NetCDF files * All metadata (in particular attributes) is stored in JSON files for the Zarr format with the following implications: * JSON does not distinguish between integers and real numbers. They are all considered as generic numbers. Whole numbers are loaded as `Int64` and real numbers `Float64`. It is not possible to store the number `1.0` as a real number. * The order of keys in a JSON document is undefined. It is therefore not possible to have a consistent ordering of the attributes or variables. - * The JSON standard does not allow the values NaN, +Inf, -Inf which is problematic for attributes ([zarr-python #412](https://github.com/zarr-developers/zarr-python/issues/412), - [zarr-specs #81](https://github.com/zarr-developers/zarr-specs/issues/81)). However, there is a special case for the fill-value to handle NaN, +Inf and -Inf. + * The JSON standard does not allow the values NaN, +Inf, -Inf which is problematic for attributes ([zarr-python #412](https://github.com/zarr-developers/zarr-python/issues/412), [zarr-specs #81](https://github.com/zarr-developers/zarr-specs/issues/81)). However, there is a special case for the fill-value to handle NaN, +Inf and -Inf. + * All dimensions must be associated to Zarr variables. diff --git a/src/dataset.jl b/src/dataset.jl index 5ee644b..82a97ec 100644 --- a/src/dataset.jl +++ b/src/dataset.jl @@ -66,11 +66,16 @@ CDM.maskingvalue(ds::ZarrDataset) = ds.maskingvalue ZarrDataset(f::Function,url::AbstractString,mode = "r"; maskingvalue = missing) -Open the zarr dataset at the url or path `url`. The mode can only be "r" (read-only) -or "c" (create). `ds` supports the API of the +Open the zarr dataset at the url or path `url`. The mode can only be `"r"` (read-only) +or `"c"` (create). `ds` supports the API of the [JuliaGeo/CommonDataModel.jl](https://github.com/JuliaGeo/CommonDataModel.jl). -The experimental `_omitcode` allows to work-around servers that return -a HTTP error different than 404 for missing chunks. +The experimental `_omitcode` allows to define which HTTP error code should be used +for missing chunks. For compatibility with python's Zarr, the HTTP error 403 +(permission denied) is also used to missing chunks in addition to 404 (not +found). + +The parameter `maskingvalue` allows to define which special value should be used +as replacement for fill values. The default is `missing`. Example: @@ -101,7 +106,6 @@ zos1 = ZarrDataset(url) do ds ds["zos"][:,:,end,1] end # implicit call to close(ds) ``` - """ function ZarrDataset(url::AbstractString,mode = "r"; parentdataset = nothing, diff --git a/src/variable.jl b/src/variable.jl index b585e9d..489380e 100644 --- a/src/variable.jl +++ b/src/variable.jl @@ -58,6 +58,7 @@ function CDM.defVar(ds::ZarrDataset,name::SymbolOrString,vtype::DataType,dimensi if isnothing(chunksizes) chunksizes = _size end + zarray = zcreate( vtype, ds.zgroup, name, _size...; chunks = chunksizes,