GH-45185: [C++][Parquet] Raise an error for invalid repetition levels when delimiting records #45186
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
Component: Parquet
Component: C++
awaiting review
|
Note for others taking a look, tests won't be successful until we merge the parquet-testing PR |
adamreeve
force-pushed
the
validate-repetition
branch
from
be347da to
22794e2
Compare
|
The failing tests all look to be caused by #45305 rather than this change. |
mapleFU
reviewed
Jan 20, 2025
github-actions
bot
added
awaiting committer review
mapleFU
reviewed
Jan 21, 2025
| TEST(TestArrowReaderAdHoc, InvalidRepetitionLevels) { | ||
| // GH-45185 - Repetition levels start with 1 instead of 0 | ||
| auto path = test::get_data_file("ARROW-GH-45185.parquet", /*is_good=*/false); | ||
| TryReadDataFile(path, ::arrow::StatusCode::IOError); |
There was a problem hiding this comment.
Would you mind also check the status is "The repetition level at the start of a record must be 0 but got ..."
mapleFU
reviewed
Jan 21, 2025
cpp/src/parquet/column_reader.cc
Outdated
| @@ -1611,7 +1611,12 @@ class TypedRecordReader : public TypedColumnReaderImpl<DType>, | |||
| // another record start or exhausting the ColumnChunk | |||
| int64_t level = levels_position_; | |||
| if (at_record_start_) { | |||
| ARROW_DCHECK_EQ(0, rep_levels[levels_position_]); | |||
| if (rep_levels[levels_position_] != 0) { | |||
There was a problem hiding this comment.
Can we use ARROW_PREDICT_FALSE to check it?
mapleFU
approved these changes
Jan 21, 2025
|
Oh my I found this is not merged...I'll rebase and try to merge this |
|
After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 85779a4. None of the specified runs were found on the Conbench server. The full Conbench report has more details. |
|
Thanks! |
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this pull request
Apr 15, 2025
…levels when delimiting records (apache#45186) ### Rationale for this change See apache#45185. Invalid repetition levels would previously only cause a fatal error in debug builds. ### What changes are included in this PR? Converts an existing `ARROW_DCHECK_EQ` of the repetition level with a check that will raise an exception in release builds too. ### Are these changes tested? Yes, using a new example file (apache/parquet-testing#67) ### Are there any user-facing changes? Yes, reading columns with invalid repetition levels as Arrow arrays will now raise an exception. * GitHub Issue: apache#45185 Lead-authored-by: Adam Reeve <[email protected]> Co-authored-by: mwish <[email protected]> Signed-off-by: mwish <[email protected]>
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this pull request
Apr 15, 2025
…levels when delimiting records (apache#45186) ### Rationale for this change See apache#45185. Invalid repetition levels would previously only cause a fatal error in debug builds. ### What changes are included in this PR? Converts an existing `ARROW_DCHECK_EQ` of the repetition level with a check that will raise an exception in release builds too. ### Are these changes tested? Yes, using a new example file (apache/parquet-testing#67) ### Are there any user-facing changes? Yes, reading columns with invalid repetition levels as Arrow arrays will now raise an exception. * GitHub Issue: apache#45185 Lead-authored-by: Adam Reeve <[email protected]> Co-authored-by: mwish <[email protected]> Signed-off-by: mwish <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
See #45185. Invalid repetition levels would previously only cause a fatal error in debug builds.
What changes are included in this PR?
Converts an existing
ARROW_DCHECK_EQof the repetition level with a check that will raise an exception in release builds too.Are these changes tested?
Yes, using a new example file (apache/parquet-testing#67)
Are there any user-facing changes?
Yes, reading columns with invalid repetition levels as Arrow arrays will now raise an exception.