Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise decimal casting for infallible conversions #7021

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

aweltsch
Copy link

@aweltsch aweltsch commented Jan 25, 2025

Which issue does this PR close?

Rationale for this change

As mentioned in the issue there are certain conversions for which we don't need to check precision / overflow and can thus improve performance by using the infallible unary kernel.

Explanation of the optimisation:

Case 1, increasing scale:

Increasing the scale will result in multiplication with powers of 10, thus being equivalent to a "shift left" of the digits.
Every number will thus "gain" additional digits. The operation is safe when the output type gives enough precision (i.e. digits) to contain the original digits, as well as the digits gained.
This can be boiled down to the condition input_prec + (output_scale - input_scale) <= output_prec

Example:

consider a conversion from Decimal(5, 0) to Decimal(8, 3) with the number 12345 * 10^0 = 12345000 * 10^(-3)
If we are starting with any number xxxxx, then an increase of scale by 3 will have the following effect on the representation: [xxxxx] -> [xxxxx000].
So for this to work, every output type needs to have at least 8 digits of precision.

Case 2, decreasing scale:

Decreasing the scale will result in division with powers of 10, thus "shifting right" of the digits and adding a rounding term. We usually need less precision to represent the result. By shifting right we lose input_scale - output_scale digits, but adding a rounding term can result in one additional digit being required, therefore we can boil this down to
input_prec - (input_scale - output_scale) < output_prec

Example:

consider a conversion from Decimal(5, 0) to Decimal(3, -3) with the number 99900 * 10^0 = 99.9 * 10^(3) which is then rounded to 100 * 10^3 = 100000
If we are starting with any number xxxxx, then and decrease the scale by 3 will have the following effect on the representation: [xxxxx] -> [xx] (+ 1 possibly). In the example plus one adds an additional digit, so the conversion to work, every output type needs to have at least 3 digits of precision.
A conversion to Decimal(2, -3) would not be possible.

Performance impact

The only cases affected are between decimal128 types, for increasing scale there is a considerable improvements that I measured of around 80%.
For decreasing the scale there was an improvement of around 25%.

cast decimal128 to decimal128 512                                  6.98      2.8±0.00µs        ? ?/sec    1.00    402.4±0.35ns        ? ?/sec
cast decimal128 to decimal128 512 lower precision                  1.01      3.0±0.00µs        ? ?/sec    1.00      2.9±0.01µs        ? ?/sec
cast decimal128 to decimal128 512 with lower scale (infallible)    1.31      3.3±0.00µs        ? ?/sec    1.00      2.5±0.00µs        ? ?/sec
cast decimal128 to decimal128 512 with same scale                  1.07     54.2±1.47ns        ? ?/sec    1.00     50.5±1.41ns        ? ?/sec
cast decimal128 to decimal256 512                                  1.00      6.1±0.01µs        ? ?/sec    1.00      6.1±0.01µs        ? ?/sec
cast decimal256 to decimal128 512                                  1.02     13.2±0.67µs        ? ?/sec    1.00     12.9±0.12µs        ? ?/sec
cast decimal256 to decimal256 512                                  1.01      6.1±0.02µs        ? ?/sec    1.00      6.1±0.03µs        ? ?/sec
cast decimal256 to decimal256 512 with same scale                  1.05     54.1±1.98ns        ? ?/sec    1.00     51.6±1.15ns        ? ?/sec

What changes are included in this PR?

I've added a new specialization for dealing with "safe" casts between the same decimal type both when reducing and increasing scale
A new benchmark for reducing scale

Are there any user-facing changes?

No, the behavior of the casts should stay the same.

@github-actions github-actions bot added the arrow Changes to the arrow crate label Jan 25, 2025
@alamb
Copy link
Contributor

alamb commented Jan 25, 2025

Thank @aweltsch -- I started the CI on this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimise Decimal Casting
2 participants