Skip to content

Reading parquet with (pre-release) arrow fails with "out of order projection is not supported" #2543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alamb opened this issue May 16, 2022 · 1 comment
Assignees
Labels
bug Something isn't working datafusion Changes in the datafusion crate

Comments

@alamb
Copy link
Contributor

alamb commented May 16, 2022

Describe the bug

When testing using a pre-release version of arrow-rs with apache/arrow-rs#1682 from @tustvold which added stricter error checking two datafusion tests fail with "out of order projection is not supported"

---- physical_plan::file_format::parquet::tests::evolved_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_filter' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:968:14

---- physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_inconsistent_order' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(ParquetError(General("out of order projection is not supported"))))', datafusion/core/src/physical_plan/file_format/parquet.rs:819:14

To Reproduce
See #2530

Expected behavior
Tests should succeed

Additional context
@tustvold suggests there is some interplay of the parquet schema adapter logic that was silently masking a bug in arrow-rs. Now that arrow-rs has stricter error checking, the query is erroring #2530 (comment)

@alamb
Copy link
Contributor Author

alamb commented Jul 29, 2022

This has since been fixed

@alamb alamb closed this as completed Jul 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working datafusion Changes in the datafusion crate
Projects
None yet
Development

No branches or pull requests

2 participants