Reduce memory usage in delta format code paths #723
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
from
0a141b3 to
ade4ee3
Compare
PatrickJin-db
changed the title
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
2 times, most recently
from
d2748d0 to
d99d480
Compare
PatrickJin-db
changed the title
PatrickJin-db
force-pushed
the
PatrickJin-db/kernel-explicit-iterator-to_pandas-args
branch
from
d99d480 to
09e1a38
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduces memory consumption when using delta-kernel-rs by converting each batch into a pandas dataframe and concatenating the results together, rather than creating a pyarrow table from all the batches and converting that table with all the results. The pyarrow table usually remains in memory during the conversion to pandas, so it is best to convert smaller batches to pandas rather than the entire table.
Existing tests should be sufficient.