Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for compressed and filtered datasets on S3 storage #120

Closed
markgoddard opened this issue Jul 6, 2023 · 0 comments · Fixed by #136
Closed

Add support for compressed and filtered datasets on S3 storage #120

markgoddard opened this issue Jul 6, 2023 · 0 comments · Fixed by #136
Assignees
@markgoddard
Labels
enhancement New feature or request

Comments

@markgoddard
Copy link

markgoddard commented Jul 6, 2023 •
edited by valeriupredoi
Loading

Currently if a dataset is compressed or has any filters applied, PyActiveStorage will raise NotImplementedError in s3.py when using the S3 storage type.

There are many compression and filter algorithms supported by netCDF/HDF5, but this task covers supporting and testing the Zlib compression algorithm as well as the HDF5 Shuffle filter. This task covers support for S3 storage, building on the local storage support discussed in #118 and added in #119
@markgoddard markgoddard added the enhancement New feature or request label Jul 6, 2023
@markgoddard markgoddard self-assigned this Jul 6, 2023
This was referenced Aug 15, 2023
Closed
Merged
markgoddard added a commit that referenced this issue Aug 15, 2023
@markgoddard
Support GZip/Zlib compression and Shuffle filter for S3
1d4b67e 
Adds support for netCDF files with Gzip/Zlib compression and/or the byte
shuffle filter when using the S3 storage backend.

The compression and filter integration tests now run when testing
against the S3 storage backend.

Closes #120
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@markgoddard markgoddard

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support GZip/Zlib compression and Shuffle filter for S3 NCAS-CMS/PyActiveStorage
1 participant
@markgoddard