Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return TableProviderFilterPushDown::Exact when Parquet Pushdown Enabled #12135

Merged
merged 24 commits into from
Sep 17, 2024
Merged

Return TableProviderFilterPushDown::Exact when Parquet Pushdown Enabled #12135

merged 24 commits into from
Sep 17, 2024
+662 −297

Commits on Sep 13, 2024

  1. feat: Preemptively filter for pushdown-preventing columns in ListingT… 

    …able
    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    4e29e20 View commit details
    Browse the repository at this point in the history

  2. Fix behavior to make all previous tests work and lay groundwork for f… 

    …uture tests
    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    762f397 View commit details
    Browse the repository at this point in the history

  3. fix: Add some more tests and fix small issue with pushdown specificity

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    30f75e0 View commit details
    Browse the repository at this point in the history

  4. test: Revive unneccesarily removed test

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d76fea3 View commit details
    Browse the repository at this point in the history

  5. ci: Fix CI issues with different combinations of exprs

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    187a121 View commit details
    Browse the repository at this point in the history

  6. fix: run fmt

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    11c62dc View commit details
    Browse the repository at this point in the history

  7. Fix doc publicity issues

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    fb53778 View commit details
    Browse the repository at this point in the history

  8. Add ::new fn for PushdownChecker

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    539b2e8 View commit details
    Browse the repository at this point in the history

  9. Remove unnecessary 'pub' qualifier

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    e13b89f View commit details
    Browse the repository at this point in the history

  10. Fix naming and doc comment of non_pushdown_columns to reflect what it… 

    … actually does (the opposite) and add back useful comments
    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5e86c0b View commit details
    Browse the repository at this point in the history

  11. fmt

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    6106ecd View commit details
    Browse the repository at this point in the history

  12. Extend FileFormat trait to allow library users to define formats whic… 

    …h support pushdown
    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    c7d6211 View commit details
    Browse the repository at this point in the history

  13. fmt

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    ba463b5 View commit details
    Browse the repository at this point in the history

  14. fix: reference real fn in doc to fix CI

    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    89f423a View commit details
    Browse the repository at this point in the history

  15. Minor: Add tests for using FilterExec when parquet was pushed down

    @alamb @itsjunetime
    alamb authored and itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    53b7046 View commit details
    Browse the repository at this point in the history

  16. Update datafusion/core/src/datasource/file_format/mod.rs

    @alamb @itsjunetime
    alamb authored and itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    5c29552 View commit details
    Browse the repository at this point in the history

  17. Pipe schema information through to TableScan and ParquetExec to facil… 

    …itate unnecessary FilterExec removal
    @itsjunetime
    itsjunetime committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    ef0affe View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. - Remove collect::<(_, _)> to satisfy msrv 

    - Remove expect(_) attr to satisfy msrv
- Update comments with more accurate details and explanations
    @itsjunetime
    itsjunetime committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    b1ee813 View commit details
    Browse the repository at this point in the history

  2. Add more details in comments for map_partial_batch 

    Co-authored-by: Andrew Lamb <[email protected]>
    @itsjunetime @alamb
    itsjunetime and alamb authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    f8adb43 View commit details
    Browse the repository at this point in the history

  3. Remove reference to issue apache#4028 as it will be closed

    @itsjunetime
    itsjunetime committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ec5f21b View commit details
    Browse the repository at this point in the history

  4. Convert normal comments to doc-comments 

    Co-authored-by: Andrew Lamb <[email protected]>
    @itsjunetime @alamb
    itsjunetime and alamb authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    19f4310 View commit details
    Browse the repository at this point in the history

  5. Clarify meaning of word projected in comment 

    Co-authored-by: Andrew Lamb <[email protected]>
    @itsjunetime @alamb
    itsjunetime and alamb authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    9cfbb5d View commit details
    Browse the repository at this point in the history

  6. Clarify more how table_schema is used differently from `projected_t… 

    …able_schema`

Co-authored-by: Andrew Lamb <[email protected]>
    @itsjunetime @alamb
    itsjunetime and alamb authored Sep 16, 2024
    Configuration menu
    Copy the full SHA
    832f7e2 View commit details
    Browse the repository at this point in the history

  7. Finish partially-written comment about SchemaMapping struct

    @itsjunetime
    itsjunetime committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ca26f03 View commit details
    Browse the repository at this point in the history