Skip to content

Commit

Permalink
more example in the docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Alexander-Barth committed Apr 30, 2024
1 parent 9434153 commit b173374
Show file tree
Hide file tree
Showing 3 changed files with 54 additions and 7 deletions.
46 changes: 44 additions & 2 deletions docs/src/index.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,57 @@

## ZarrDatasets


See the [documentation of JuliaGeo/CommonDataModel.jl](https://juliageo.org/CommonDataModel.jl/stable/) for the full documentation of the API. As a quick reference, here is an example how to create and read a Zarr file store as a quick reference.

### Create a Zarr file store

The following example create a Zarr file store in the directory `"/tmp/test-zarr"`:

```julia
using ZarrDatasets

# sample data
data = [i+j for i = 1:3, j = 1:5]

directoryname = "/tmp/test-zarr4"
mkdir(directoryname)

ds = ZarrDataset(directoryname,"c")
defDim(ds,"lon",size(data,1))
defDim(ds,"lat",size(data,2))
zv = defVar(ds,"varname",Int64,("lon","lat"))
zv[:,:] = data
zv.attrib["units"] = "m"
close(ds)
```

### Loading a Zarr file store

The data and units can be loaded by indexing the data set structure `ds`.

```julia
using ZarrDatasets
directoryname = "/tmp/test-zarr4"
ds = ZarrDataset(directoryname)
data = ds["varname"][:,:]
data_units = ds["varname"].attrib["units"]
```



```@autodocs
Modules = [ZarrDatasets]
```





### Differences between Zarr and NetCDF files

* All metadata (in particular attributes) is stored in JSON files for the Zarr format with the following implications:
* JSON does not distinguish between integers and real numbers. They are all considered as generic numbers. Whole numbers are loaded as `Int64` and real numbers `Float64`. It is not possible to store the number `1.0` as a real number.
* The order of keys in a JSON document is undefined. It is therefore not possible to have a consistent ordering of the attributes or variables.
* The JSON standard does not allow the values NaN, +Inf, -Inf which is problematic for attributes ([zarr-python #412](https://github.com/zarr-developers/zarr-python/issues/412),
[zarr-specs #81](https://github.com/zarr-developers/zarr-specs/issues/81)). However, there is a special case for the fill-value to handle NaN, +Inf and -Inf.
* The JSON standard does not allow the values NaN, +Inf, -Inf which is problematic for attributes ([zarr-python #412](https://github.com/zarr-developers/zarr-python/issues/412), [zarr-specs #81](https://github.com/zarr-developers/zarr-specs/issues/81)). However, there is a special case for the fill-value to handle NaN, +Inf and -Inf.
* All dimensions must be associated to Zarr variables.
14 changes: 9 additions & 5 deletions src/dataset.jl
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,16 @@ CDM.maskingvalue(ds::ZarrDataset) = ds.maskingvalue
ZarrDataset(f::Function,url::AbstractString,mode = "r";
maskingvalue = missing)
Open the zarr dataset at the url or path `url`. The mode can only be "r" (read-only)
or "c" (create). `ds` supports the API of the
Open the zarr dataset at the url or path `url`. The mode can only be `"r"` (read-only)
or `"c"` (create). `ds` supports the API of the
[JuliaGeo/CommonDataModel.jl](https://github.com/JuliaGeo/CommonDataModel.jl).
The experimental `_omitcode` allows to work-around servers that return
a HTTP error different than 404 for missing chunks.
The experimental `_omitcode` allows to define which HTTP error code should be used
for missing chunks. For compatibility with python's Zarr, the HTTP error 403
(permission denied) is also used to missing chunks in addition to 404 (not
found).
The parameter `maskingvalue` allows to define which special value should be used
as replacement for fill values. The default is `missing`.
Example:
Expand Down Expand Up @@ -101,7 +106,6 @@ zos1 = ZarrDataset(url) do ds
ds["zos"][:,:,end,1]
end # implicit call to close(ds)
```
"""
function ZarrDataset(url::AbstractString,mode = "r";
parentdataset = nothing,
Expand Down
1 change: 1 addition & 0 deletions src/variable.jl
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ function CDM.defVar(ds::ZarrDataset,name::SymbolOrString,vtype::DataType,dimensi
if isnothing(chunksizes)
chunksizes = _size
end

zarray = zcreate(
vtype, ds.zgroup, name, _size...;
chunks = chunksizes,
Expand Down

0 comments on commit b173374

Please sign in to comment.