fix: Fix performance regression for sort/gather on list/array columns #19564
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #19559
It turns out, our slow-paths for
take_unchecked
are very slow, because the implementation constructs aGrowable
with aVec
containing 1 unit-lengthArray
per gather indexpolars/crates/polars-arrow/src/compute/take/list.rs
Lines 35 to 49 in ebeeea7
And all of them end up being bit-counted in
Growable::new
:Ppolars/crates/polars-arrow/src/array/growable/list.rs
Line 49 in ebeeea7
Also, the reason why we regressed in list-gather performance was because we were previously not using that super-slow list gather implementation until a dispatch
impl ChunkTakeUnchecked<IdxCa> for ListChunked {
was added to it in #19327