Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAVLinkParam: fix rounding to 7 digits #3389

Merged
merged 1 commit into from
Aug 15, 2024

Conversation

robertlong13
Copy link
Collaborator

Rounding to a fixed number of decimal places leads to over rounding small numbers and under-rounding big numbers. I first encountered this problem when dealing parameter values in the range of 10^-6 that kept getting way rounded off. That's not a common order of magnitude for a parameter in ArduPilot, but we shouldn't rule it out (in my case, they were part of a lua script; I don't think any official params are in this range).

Some samples from the "Immediate Window" in Visual Studio

(double)0.8f
0.800000011920929
Math.Round((double)0.8f, 7)
0.8
RoundToSignificantDigits((double)0.8f, 7)
0.8

(double)100.8f
100.80000305175781
Math.Round((double)100.8f, 7)
100.8000031
RoundToSignificantDigits((double)100.8f, 7)
100.8

(double)8e-9f
7.9999997737445483E-09
Math.Round((double)8e-9f, 7)
0
RoundToSignificantDigits((double)8e-9f, 7)
8E-09

Now, another thing to consider, do we actually need to round at all, or should we just be casting to double and be done with it? The only application I could think of where this rounding was important was string conversion, but we already correctly handle that in the ToString method.

Rounding to fixed number of decimal places led to over rounding small
numbers and under-rounding big numbers.
@meee1 meee1 merged commit 0bb3a2d into ArduPilot:master Aug 15, 2024
2 of 6 checks passed
@robertlong13 robertlong13 deleted the pr/param_rounding_fix branch August 15, 2024 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants