Skip to content

Upgrade to arrow 15 #2631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 31, 2022
Merged

Upgrade to arrow 15 #2631

merged 5 commits into from
May 31, 2022

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented May 27, 2022

Which issue does this PR close?

Closes #2530

Closes #2453

Rationale for this change

Arrow 15 contains breaking changes, lets validate them prior to release.

What changes are included in this PR?

Updates to the current arrow-rs master

Are there any user-facing changes?

No

Does this PR break compatibility with Ballista?

Possibly, I think I need to modify ballista to use the re-exports from Datafusion

@tustvold tustvold force-pushed the update-to-latest-arrow branch from c09bf2a to 0d7b45f Compare May 27, 2022 10:58
@github-actions github-actions bot added the physical-expr Changes to the physical-expr crates label May 27, 2022
@@ -365,7 +365,7 @@ mod tests {
let expected = Statistics {
is_exact: true,
num_rows: Some(3),
total_byte_size: Some(416), // this might change a bit if the way we compute the size changes
total_byte_size: Some(464), // this might change a bit if the way we compute the size changes
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes to DataType have made it bigger, I don't think this is a problem

@alamb
Copy link
Contributor

alamb commented May 30, 2022

FWIW arrow 15 is now released apache/arrow-rs#1727

@github-actions github-actions bot added logical-expr Logical plan and expressions sql SQL Planner labels May 30, 2022
@github-actions github-actions bot added the development-process Related to development process of DataFusion label May 30, 2022
@tustvold tustvold marked this pull request as ready for review May 30, 2022 15:53
@tustvold
Copy link
Contributor Author

Currently blocked on apache/datafusion-ballista#48

@tustvold tustvold changed the title Prepare for arrow 15 release Upgrade to arrow 15 May 30, 2022
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tustvold

@alamb
Copy link
Contributor

alamb commented May 31, 2022

🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate datafusion Changes in the datafusion crate development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sql SQL Planner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect Parquet Projection For Nested Types
3 participants