Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rechunk to an existing store #148

Open
jbeezley opened this issue Nov 29, 2023 · 0 comments
Open

Rechunk to an existing store #148

jbeezley opened this issue Nov 29, 2023 · 0 comments

Comments

@jbeezley
Copy link

I have an existing data pipeline where I have data coming in incrementally. I have an existing pipeline performing a naive rechunking to a zarr store whenever new data comes into the source store. Rechunker has a much better algorithm I would like to use, but it doesn't have the ability to target an existing store.

This problem seems related to #8 however, for my use case a simpler implementation would be to optionally skip the call at https://github.com/pangeo-data/rechunker/blob/master/rechunker/api.py#L599 and open the dataset instead.

I would be willing to implement this via an optional kwarg, but I wanted to check if such a change would be accepted or if there are any issues with it that I'm not considering. Clearly, there could be problems if the dimensions/variables of the destination are not compatible. I could check that after opening or just let the exceptions from zarr pass through. Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant