MAVLinkParam: fix rounding to 7 digits #3389
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rounding to a fixed number of decimal places leads to over rounding small numbers and under-rounding big numbers. I first encountered this problem when dealing parameter values in the range of 10^-6 that kept getting way rounded off. That's not a common order of magnitude for a parameter in ArduPilot, but we shouldn't rule it out (in my case, they were part of a lua script; I don't think any official params are in this range).
Some samples from the "Immediate Window" in Visual Studio
Now, another thing to consider, do we actually need to round at all, or should we just be casting to double and be done with it? The only application I could think of where this rounding was important was string conversion, but we already correctly handle that in the
ToString
method.