Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update backends tutorials intro #266

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5e2045b
add draft chart to markdown file
JessicaS11 Jun 10, 2024
3c65328
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 10, 2024
5c803c2
add mermaid sphinx extension
JessicaS11 Jun 11, 2024
67aa3f6
revise backend content
JessicaS11 Jun 11, 2024
42aa2ad
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 11, 2024
3bae4fc
add some color styling
JessicaS11 Jun 11, 2024
06da29a
add sphinx mermaid package to env
JessicaS11 Jun 11, 2024
21ed978
save conflict changes
JessicaS11 Jun 11, 2024
bbcb478
Merge branch 'main' into engine-flow
JessicaS11 Jun 11, 2024
f28966f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 11, 2024
53659c9
update conda lock files
JessicaS11 Jun 11, 2024
bbe08f9
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 11, 2024
7d74f63
add myst parser to use mermaid lexer with sphinx
JessicaS11 Jun 12, 2024
c7e1adf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 12, 2024
bac54de
fix dash to underscore
JessicaS11 Jun 12, 2024
45de572
try changing order of extensions
JessicaS11 Jun 12, 2024
8a6a28f
add source_suffix sphinx config
JessicaS11 Jun 12, 2024
44c8d64
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 12, 2024
746f9f0
Merge branch 'main' into engine-flow
JessicaS11 Jun 20, 2024
382a532
add curly braces
JessicaS11 Jun 20, 2024
57acd92
remove diagram (moved to docs) and update text
JessicaS11 Jun 25, 2024
da9e806
revert environment changes to main
JessicaS11 Jun 25, 2024
5870819
Merge branch 'main' into engine-flow
JessicaS11 Jun 25, 2024
0857457
add h5netcdf to list of backends
JessicaS11 Sep 9, 2024
daa22e6
Merge branch 'main' into engine-flow
JessicaS11 Sep 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 29 additions & 24 deletions advanced/backends/backends.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,49 @@
# Creating new backends
# Reading data using backends

## Introduction

You can [read different type of files](https://docs.xarray.dev/en/stable/user-guide/io.html)) in `xr.open_dataset` by specifying the engine to be used:
You can [read different types of files](https://docs.xarray.dev/en/stable/user-guide/io.html) in `xr.open_dataset` by specifying the engine to be used:

```python
import xarray as xr
xr.open_dataset("my_file.grib" , engine="cfgrib")
```

For each available engine there is an underlying backend, that reads the data and pack them in a dataset.
Navigating Xarray backends can be confusing,
so we recommend checking out [this flow chart](https://docs.xarray.dev/en/stable/user-guide/io.html)
to help you figure out which engine you need and how to use it.
JessicaS11 marked this conversation as resolved.
Show resolved Hide resolved

Xarray bundles several backends internally for the following formats:
You can see what backends are currently available in your working environment
with `xarray.backends.list_engines()`.

- netcdf4 - netCDF4
- scipy - netCDF3
- zarr - Zarr
- pydap - DAP
- ...
## Why use the Xarray backend API to write your own backend?

External Backends that use the new backend API (xarray >= v0.18.0) that allows to add support for backend without any change to Xarray
- Your users don't need to learn a new interface; they can use `xr.open_dataset` with the `engine` kwarg.
- With little extra effort you can have lazy loading with Dask. Simply implement a function for reading blocks and Xarray will manage lazy loading with Dask for you.
- It's easy to implement: using the backend API (introduced in v0.18.0), you don't need to integrate any code in Xarray.

- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL)
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE
- ...
## More Information

## Why using the Xarray backend API
See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more details on adding and registering a new backend.

- Your users don't need to learn a new interface that is they can use `xr.open_dataset` with `engine` kwarg.
- With little extra effort you can have lazy loading with Dask. you have to implement a function for reading blocks and Xarray will manage lazy loading with Dask for you
- It's easy to implement: you don't need to integrate any code in Xarray
Follow the tutorials on creating a new backend for binary files.

```{tableofcontents}

## Next
```

See the [documentation](https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html) for more.
### Links to internal backends

Follow the tutorial on creating a new backend for binary files.
- [netcdf4](https://pypi.org/project/netCDF4/) - netCDF4
- [scipy](https://scipy.org/) - netCDF3
- [zarr](https://pypi.org/project/zarr/) - Zarr
- [pydap](https://pydap.github.io/pydap/) - Data Access Protocol (DAP/DODS/OPeNDAP)
- [h5netcdf](https://h5netcdf.org/) - hdf5
JessicaS11 marked this conversation as resolved.
Show resolved Hide resolved

```{tableofcontents}
### Links to external backends (not comprehensive)

```
- [cfgrib](https://github.com/ecmwf/cfgrib) - GRIB
- [tiledb](https://github.com/TileDB-Inc/TileDB-CF-Py) - TileDB
- [rioxarray](https://corteva.github.io/rioxarray/stable/) - GeoTIFF, JPEG-2000, ESRI-hdr, etc (via GDAL)
- [xarray-sentinel](https://github.com/bopen/xarray-sentinel) - Sentinel-1 SAFE
- ...
Loading