- J. Emmanuel Johnson
- Quentin Febvre
This is a lightweight package to create patch items from xarray data structures. This makes it more compatible with machine learning datasets and dataloaders like PyTorch or TensorFlow. The user simply needs to define the patch dimensions and the stride dimensions and you are good to go! It also reconstructs (or unpatchifies) from arbitrary patches which allows for more robust inference procedures, e.g. to account for border effects from CNN models.
import xarray as xr
from xrpatcher import XRDAPatcher
# load demo dataset
data = xr.tutorial.load_dataset("eraint_uvz")
# extract demo dataarray
data = data.u[..., :240, :360]
# Instantiate the patching logic for training
patches = dict(longitude=30, latitude=30)
train_patcher = XRDAPatcher(
da=data,
patches=patches,
strides=patches, # No Overlap
check_full_scan=True # check no extra dimensions
)
# Instantiate the patching logic for testing
patches = dict(longitude=30, latitude=30)
strides = dict(longitude=5, latitude=5)
test_patcher = XRDAPatcher(
da=data,
patches=patches,
strides=strides, # Overlap
check_full_scan=True # check no extra dimensions
)Patching Crash Course
We have an extended example where we demonstrate some of the ways to do the reconstruction!
PyTorch Integration
We have an extended example where we demonstrate some nifty PyTorch Integration.
Example 3: PyTorch Integration Concatenate Multiple domain
We demonstrate in this example how this tool can be used to create more complex dataloading like jointly training on separate regions
We can directly install it via pip from the
pip install xrpatcherWe can also clone the git repository
git clone https://github.com/jejjohnson/xrpatcher.git
cd xrpatcherWe use conda/mamba as our package manager. To install from the provided environment files run the following command.
mamba env create -n environment.yamlThe easiest way to get started is to simply use the poetry package which installs all necessary dev packages as well
poetry installWe can also install via pip as well
pip install .There are a few other packages that gave us inspiration for this.