Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update readme and slides #22

Merged
merged 3 commits into from
Oct 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 3 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
## Overview

This a work-in-progress guidebook on existing warp resampling / reprojection methods in Python, along with some memory and statistical wall-time profiling results.

## Caveat

This is a WIP guidebook. We are presenting early on in the development to guide discussions and future work. The output from different resampling methods has not yet been verified, important parameters (e.g., target no_data values) have not been set, and not all methods have been implemented. We encourage people to contribute by building on the notebooks in the `examples` directory or participating in discussions on this repo or on the Pangeo discourse.
This repository and the accompanying Quarto book contains examples of using warp resampling / reprojection methods in Python, along with memory and statistical wall-time profiling results.

## Memory and time profiling

Resampling and reprojection (i.e., warp resampling) are essential steps for generating raster tiles for browser based visualization. Further, warp resampling is often one of the most time consuming and memory intensive portions of the tile generation process. The importance and complexity of this step motivates an exploration of different warp resampling options.

### Goals

Compare memory and time performance for generating a zoom level 0 256 x 256 raster from one timestep and variable of the MUR SST dataset using the following approaches:
Compare memory and time performance for generating Web Mercator Quad Tree raster tiles from one timestep and variable of the MUR SST and GPM IMERG dataset using the following approaches:

- [osgeo.warp](https://gdal.org/en/latest/api/python/utilities.html#osgeo.gdal.Warp)
- [rasterio.warp.reproject](https://rasterio.readthedocs.io/en/stable/api/rasterio.warp.html#rasterio.warp.reproject)
Expand All @@ -33,14 +29,7 @@ Out-of-scope:
- [pygmt.grdproject](https://www.pygmt.org/latest/api/generated/pygmt.grdproject.html#pygmt.grdproject) - [web mercator not amongst supported projections](https://www.pygmt.org/latest/projections/index.html)
- [verde](https://www.fatiando.org/verde/latest/) - not used for raster -> raster resampling (only points -> raster)

These methods will be run on the full resolution dataset. Nearest neighbor interpolation will be used for the first comparison. For simplicity, the amount of time necessary to generate a resampled array and the maximum amount of heap memory allocated will be measured.

### Planned extensions

- Compare to results when using a 2x and 4x downsampled versions to better understand the time and memory complexity
- Compare to results when using a virtual dataset (e.g., VRT, Kerchunk reference file).
- Compare results when reading from a dataset stored locally versus in cloud object storage.
- Compare to results when using a cloud-optimized dataset (Zarr).
These methods will be run on the full resolution dataset. Nearest neighbor interpolation will be used for the first comparison. For simplicity, the amount of time necessary to generate a resampled array and the maximum amount of heap memory allocated will be measured. We also compare results when using a virtual dataset (e.g., VRT, Kerchunk reference file), when reading from a dataset stored locally versus in cloud object storage, and when using a cloud-optimized dataset (Zarr.)

### Possible extensions

Expand Down
9 changes: 8 additions & 1 deletion examples/generate-weights.ipynb
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pre-generate weights for resampling with XESMF or sparse"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -265,7 +272,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.10"
"version": "3.12.6"
}
},
"nbformat": 4,
Expand Down
8 changes: 4 additions & 4 deletions pages/slides.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -96,14 +96,14 @@ Visualization

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/earthdata-download.html"}

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/resample-gdal-netcdf-vsis3.html"}
## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/resample-netcdf-rasterio-netcdf-local.html"}

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/resample-odc-h5netcdf-.html"}

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/future-resample-xesmf-h5netcdf-.html"}
## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/generate-weights.html"}

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/process-results.html"}

## {background-iframe="https://developmentseed.org/warp-resample-profiling/examples/process-gpm-results.html"}

## {background-iframe="https://link.excalidraw.com/readonly/GgF1qYswczuP9WKpoCvU"}

## Observations and opinions
Expand Down