Improve feature flag CI coverage datafusion and datafusion-functions
#15203
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
development-process
alamb
commented
Mar 13, 2025
.github/workflows/rust.yml
Outdated
| @@ -82,6 +82,8 @@ jobs: | |||
| uses: ./.github/actions/setup-builder | |||
| with: | |||
| rust-version: stable | |||
| - name: Check datafusion-common (default features) | |||
There was a problem hiding this comment.
there was no check for default set of features (even the tests run with avro, etc) so I added that
alamb
changed the title
datafusion and datafusion-functionsdatafusion and datafusion-functions, remove core_expressions feature
alamb
changed the title
datafusion and datafusion-functions, remove core_expressions featuredatafusion and datafusion-functions
alamb
commented
Mar 13, 2025
.github/workflows/rust.yml
Outdated
| - name: Check datafusion-substrait (no-default-features) | ||
| run: cargo check --profile ci --all-targets --no-default-features -p datafusion-substrait | ||
| # | ||
| # Note: Only check libraries (not --all-targets) to cover end user APIs |
There was a problem hiding this comment.
The idea is to test only --all-targets for the default features (what a developer would likely run), otherwise the CI will just verify that a user of the crate can compile things rather than check all tests and benches too
|
@xudong963 I wonder if you have some time to review this PR? |
|
Thank you for the review @xudong963 -- much apprecaited |
alamb
added a commit
to alamb/datafusion
that referenced
this pull request
Mar 17, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 27, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 28, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 29, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 29, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 31, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Mar 31, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Apr 7, 2025
acking-you
pushed a commit
to acking-you/arrow-datafusion
that referenced
this pull request
Apr 7, 2025
alamb
added a commit
that referenced
this pull request
Apr 8, 2025
* [draft] add shot circuit in BinaryExpr * refactor: add check_short_circuit function * refactor: change if condition to match * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * add extend sql & docs * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * fix: incorrect false judgment * add test * separate q6 to new PR * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (#15203) * add benchmark for boolean_op * fix cargo doc * add binary_op bench * Better comments --------- Co-authored-by: Kristin Cowalcijk <[email protected]> Co-authored-by: Tim Saucer <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
nirnayroy
pushed a commit
to nirnayroy/datafusion
that referenced
this pull request
May 2, 2025
* [draft] add shot circuit in BinaryExpr * refactor: add check_short_circuit function * refactor: change if condition to match * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * add extend sql & docs * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * fix: incorrect false judgment * add test * separate q6 to new PR * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * feat: Add support for --mem-pool-type and --memory-limit options to multiple benchmarks (apache#14642) * Add support --mem-pool-type and --memory-limit options for all benchmarks * Add --sort-spill-reservation-bytes option * Chore/Add additional FFI unit tests (apache#14802) * Add unit tests to FFI_ExecutionPlan * Add unit tests for FFI table source * Add round trip tests for volatility * Add unit tests for FFI insert op * Simplify string generation in unit test Co-authored-by: Andrew Lamb <[email protected]> * Fix drop of borrowed value --------- Co-authored-by: Andrew Lamb <[email protected]> * Improve feature flag CI coverage `datafusion` and `datafusion-functions` (apache#15203) * add benchmark for boolean_op * fix cargo doc * add binary_op bench * Better comments --------- Co-authored-by: Kristin Cowalcijk <[email protected]> Co-authored-by: Tim Saucer <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
datafusion-substraitanddatafusion-protoCI feature checks, increase coverage #15156Rationale for this change
I reviewed the CI coverage for datafusion's (many!) feature flag and it is haphazard -- for example it was not clear where to add a test for #15124
What changes are included in this PR?
datafusionanddatafusion-functionscrateAre these changes tested?
Are there any user-facing changes?