Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Input / chunk preprocessing #48

Closed
rabernat opened this issue Jan 24, 2021 · 0 comments · Fixed by #65
Closed

Input / chunk preprocessing #48

rabernat opened this issue Jan 24, 2021 · 0 comments · Fixed by #65
Labels
design question A question of the design of Pangeo Forge good first issue Good for newcomers recipe enhancement Solving this requires us to enhance the recipe classes

Comments

@rabernat
Copy link
Contributor

rabernat commented Jan 24, 2021 •
edited
Loading

It will be very important to provide hooks for recipes to plug in custom processing functions. These can be useful for fixing weird metadata or deriving new variables. Raised by @naomi-henderson in #47:

pre-processing is not available in open_dataset

We could easily add a preprocessing hook to the Recipe params here:

https://github.com/pangeo-forge/pangeo-forge/blob/f9b698a35a62ab169bce546b15fef9722c70cd90/pangeo_forge/recipe.py#L154-L155

There are actually several different points where such hooks could be called. In open_input.

https://github.com/pangeo-forge/pangeo-forge/blob/f9b698a35a62ab169bce546b15fef9722c70cd90/pangeo_forge/recipe.py#L244-L254

Or in open_chunk

https://github.com/pangeo-forge/pangeo-forge/blob/f9b698a35a62ab169bce546b15fef9722c70cd90/pangeo_forge/recipe.py#L259-L269

Maybe it would be useful to have hooks for both?
@rabernat rabernat added recipe enhancement Solving this requires us to enhance the recipe classes design question A question of the design of Pangeo Forge good first issue Good for newcomers labels Jan 24, 2021
@rabernat rabernat mentioned this issue Jan 24, 2021
Open
@rabernat rabernat mentioned this issue Jan 28, 2021
Merged
@rabernat rabernat linked a pull request Jan 28, 2021 that will close this issue
Add process_input and process_chunk hooks #65
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
design question A question of the design of Pangeo Forge good first issue Good for newcomers recipe enhancement Solving this requires us to enhance the recipe classes
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add process_input and process_chunk hooks davidbrochart/pangeo-forge
1 participant
@rabernat