PDA

View Full Version : Issue with fixPrecision in NumberEditor



barryglenn
21 Dec 2006, 7:51 PM
I think there may be a problem with the fixPrecision function in the NumberEditor. When you use Math.round as the decimalPrecisionFn, I think it may be outputting incorrect values. Here is an example which rounds .4449 to two decimal places to .45 instead of .44:

<script type="text/javascript" src="http://us.js2.yimg.com/us.js.yimg.com/lib/common/utils/2/utilities_2.1.0.js"></script>
<script type="text/javascript" src="/yui-ext.0.33/yui-ext-debug.js"></script>

<script type="text/javascript">

var editor = new YAHOO.ext.grid.NumberEditor({'decimalPrecisionFcn':Math.round,'decimalPrecision':2});
document.write('editor.fixPrecision(.4449, decimalPrecision:2) == ' + editor.fixPrecision(.4449));

</script>

jack.slocum
22 Dec 2006, 2:25 PM
The problem is with JS rounding. The Number editor tries to correct JS, which has rounding errors (big time!). This is an example of where the fix looks like it's actually creating an error of it's own. I will have a look at it.