Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MMap support for IPC files #6709

Open
fabianmurariu opened this issue Nov 9, 2024 · 2 comments
Open

MMap support for IPC files #6709

fabianmurariu opened this issue Nov 9, 2024 · 2 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@fabianmurariu
Copy link
Contributor

fabianmurariu commented Nov 9, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We are using polars-arrow and would prefer to move to arrow-rs as we are looking into using Datafusion as well

Describe the solution you'd like
We are dependent on mmap support for IPC format

Describe alternatives you've considered
keep using polars-arrow

Additional context
It looks like Buffer and Bytes have the right functions and support for Custom dealocation

@fabianmurariu fabianmurariu added the enhancement Any new improvement worthy of a entry in the changelog label Nov 9, 2024
@tustvold
Copy link
Contributor

The docs could definitely be improved, but the way to do this is to use FileDecoder with a buffer constructed from your mmap file - https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.FileDecoder.html

@fabianmurariu
Copy link
Contributor Author

I'll have a look these days, if it works out I'll raise a PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

No branches or pull requests

2 participants