Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix bug with decoding timestamp as decimal #36

Merged
merged 2 commits into from
Jan 3, 2025

Conversation

Jefffrey
Copy link
Collaborator

Closes #14

Copy link
Collaborator Author

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @progval for identifying this bug

@@ -103,7 +103,7 @@ fn decimal128_decoder(
let data = get_rle_reader(column, data)?;

let secondary = stripe.stream_map().get(column, Kind::Secondary);
let secondary = get_rle_reader(column, secondary)?;
let secondary = get_unsigned_rle_reader(column, secondary);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug was here. Secondary is unsigned stream. It's retrieved correctly for timestamp, but was mistakenly taken as signed here. See line 62 for correct behaviour.

fn get_inner_timestamp_decoder<T: ArrowTimestampType + Send>(
column: &Column,
stripe: &Stripe,
seconds_since_unix_epoch: i64,
) -> Result<PrimitiveArrayDecoder<T>> {
let data = stripe.stream_map().get(column, Kind::Data);
let data = get_rle_reader(column, data)?;
let secondary = stripe.stream_map().get(column, Kind::Secondary);
let secondary = get_unsigned_rle_reader(column, secondary);
let present = PresentDecoder::from_stripe(stripe, column);
let iter = Box::new(TimestampDecoder::<T>::new(
seconds_since_unix_epoch,
data,
secondary,
));
Ok(PrimitiveArrayDecoder::<T>::new(iter, present))
}

@Jefffrey Jefffrey requested review from progval and Xuanwo January 3, 2025 03:03
@Xuanwo Xuanwo changed the title Fix bug with decoding timestamp as decimal fix: Fix bug with decoding timestamp as decimal Jan 3, 2025
Copy link
Collaborator

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Jefffrey for fixing this.

@Xuanwo
Copy link
Collaborator

Xuanwo commented Jan 3, 2025

There is CI failure that not related to this PR, I will try fixing them later.

@Xuanwo Xuanwo merged commit 03429d6 into datafusion-contrib:main Jan 3, 2025
11 of 12 checks passed
@Jefffrey Jefffrey deleted the fix_14 branch January 3, 2025 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect Timestamp decoding into Decimal128 since v0.4.0
2 participants