Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support selection pruning #17

Open
Jefffrey opened this issue Nov 3, 2023 · 4 comments
Open

Support selection pruning #17

Jefffrey opened this issue Nov 3, 2023 · 4 comments
Labels
enhancement New feature or request medium Medium priority

Comments

@Jefffrey
Copy link
Collaborator

Jefffrey commented Nov 3, 2023

Make use of file statistics, stripe statistics, column statistics, row group indexes, and bloom filters

Need way to expose this functionality so users (like datafusion) can utilize to efficiently query large ORC files, e.g. via predicate pushdown

@Jefffrey Jefffrey changed the title Support predicate pushdown Support selection pruning Nov 4, 2023
@Jefffrey Jefffrey added enhancement New feature or request medium Medium priority labels Apr 1, 2024
@Jefffrey
Copy link
Collaborator Author

Jefffrey commented Apr 1, 2024

Take inspiration from how parquet handles exposing the necessary information/behaviour: https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/type.ParquetRecordBatchReaderBuilder.html

@waynexia waynexia transferred this issue from datafusion-contrib/datafusion-orc Oct 30, 2024
@suxiaogang223
Copy link
Contributor

I'd like to do some research for this PR.

@suxiaogang223
Copy link
Contributor

It seems that there is no implementation of predicate pushdown in arrow-rs. I think the implementation in DataFusion can be used as a reference to determine what APIs need to be provided for ORC.
https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/parquet/row_group_filter.rs

@Jefffrey
Copy link
Collaborator Author

It seems that there is no implementation of predicate pushdown in arrow-rs. I think the implementation in DataFusion can be used as a reference to determine what APIs need to be provided for ORC. https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/parquet/row_group_filter.rs

Yeah this issue is a port from when this repo was split from the datafusion code; I think it should still be applicable for having APIs in this crate that'll enable pushdown from the datafusion code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request medium Medium priority
Projects
None yet
Development

No branches or pull requests

2 participants