Add roundtrip test case for null buffer test #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jefffrey
commented
Dec 31, 2024
Comment on lines
+500
to
+534
| // Encoding to bytes | ||
| let mut f = vec![]; | ||
| let mut writer = ArrowWriterBuilder::new(&mut f, batch.schema()) | ||
| .try_build() | ||
| .unwrap(); | ||
| writer.write(&batch).unwrap(); | ||
| writer.close().unwrap(); | ||
| let mut f = Bytes::from(f); | ||
| let builder = ArrowReaderBuilder::try_new(f.clone()).unwrap(); | ||
|
|
||
| // Ensure the ORC file we wrote indeed has a present stream | ||
| let stripe = Stripe::new( | ||
| &mut f, | ||
| &builder.file_metadata, | ||
| builder.file_metadata().root_data_type(), | ||
| &builder.file_metadata().stripe_metadatas()[0], | ||
| ) | ||
| .unwrap(); | ||
| assert_eq!(stripe.columns().len(), 1); | ||
| // Make sure we're getting the right column | ||
| assert_eq!(stripe.columns()[0].name(), "int64"); | ||
| // Then check present stream | ||
| let present_stream = stripe | ||
| .stream_map() | ||
| .get_opt(&stripe.columns()[0], proto::stream::Kind::Present); | ||
| assert!(present_stream.is_some()); | ||
|
|
||
| // Decoding from bytes | ||
| let reader = builder.build(); | ||
| let rows = reader.collect::<Result<Vec<_>, _>>().unwrap(); | ||
|
|
||
| assert_eq!(rows.len(), 1); | ||
| assert_eq!(rows[0].num_columns(), 1); | ||
| // Ensure read array has no null buffer | ||
| assert!(rows[0].column(0).nulls().is_none()); |
There was a problem hiding this comment.
A bit verbose but has important checks to ensure that if our ORC writer behaviour changes (such that it no longer writes an empty present stream) we'll be informed of this test breaking
|
thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As suggested in this comment: datafusion-contrib#13 (review)
Instead of relying on PyORC to generate a file which tests the fix, we can rely on our own writing behaviour to generate a test file in the expected format.