Rounding question related to RoundNanoFractionTo
Posted: 23.05.2021, 21:47
This may be something I don't understand, but RoundNanoFractionTo seems to give some unexpected results.
If you call:
x := RoundNanoFractionTo(123456789, 3);
The result is 124000000. Am I crazy or should that be 123000000? I'm aware of different rounding schemes (banker's rounding and such) but I can't figure out this one. I see that the way it works is to get the "rest" of the digits (456789) and then it compares it to 444445 and if it's greater it adds one which is where the 124 comes from.
It seems to be that
const HalfFractModulos: array [TFractionRoundToScale] of Cardinal = ( 444444445, 44444445, 4444445, 444445, 44445, 4445, 445, 45, 5,0);
should be:
const HalfFractModulos: array [TFractionRoundToScale] of Cardinal = ( 500000000, 50000000, 5000000, 500000, 50000, 5000, 500, 50, 5,0);
-Mark
If you call:
x := RoundNanoFractionTo(123456789, 3);
The result is 124000000. Am I crazy or should that be 123000000? I'm aware of different rounding schemes (banker's rounding and such) but I can't figure out this one. I see that the way it works is to get the "rest" of the digits (456789) and then it compares it to 444445 and if it's greater it adds one which is where the 124 comes from.
It seems to be that
const HalfFractModulos: array [TFractionRoundToScale] of Cardinal = ( 444444445, 44444445, 4444445, 444445, 44445, 4445, 445, 45, 5,0);
should be:
const HalfFractModulos: array [TFractionRoundToScale] of Cardinal = ( 500000000, 50000000, 5000000, 500000, 50000, 5000, 500, 50, 5,0);
-Mark