xsd2inst incorrectly handling numeric precision
-----------------------------------------------

                 Key: XMLBEANS-410
                 URL: https://issues.apache.org/jira/browse/XMLBEANS-410
             Project: XMLBeans
          Issue Type: Bug
          Components: Tools
    Affects Versions: Version 2.4.1
         Environment: All OS, all hardware
            Reporter: Mark D Henning
             Fix For: TBD


as an example, a xs:decimal (totalDig:20, fracDig:0) was creating 1000.00  whic of course, does not validate.

The problem is in the org.apache.xmlbeans.impl.xsd2inst.SampleXmlUtil.java class.  
Root cause of the problem is that the programmer was calling the setScale() method on a BigDecimal, as though it modifies the existing BigDecimal object (i.e. BigDecimal bd = new BigDecimal("1000.00"); bd.setScale(1))  setScale does not modify the existing, but returns a new BigDecimal object.  Therefore the correct call would be (BigDecimal bd = new BigDecimal("1000.00"); bd = bd.setScale(1)).
The lines included below represent the repair of the formatDecimal() method which is the eronious function.

       // We have the number
        // Adjust the scale according to the totalDigits and fractionDigits
        int digits = 0;
        BigDecimal ONE = new BigDecimal(BigInteger.ONE);
        for (BigDecimal n = result; n.abs().compareTo(ONE) >= 0; digits++)
            n = n.movePointLeft(1);

        if (fractionDigits > 0)
            if (totalDigits >= 0)
                result = result.setScale(Math.max(fractionDigits, totalDigits - digits));
            else
                result = result.setScale(fractionDigits);
        else if (fractionDigits == 0)
            result = result.setScale(0);

        return result.toString();
    }

I would recommend that the code tree be searched for other setScale method calls to see if others need to be fixed.  I currently do not have write access to the subversion repository, so I am unable to check this fix in myself.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xmlb...
For additional commands, e-mail: dev-help@xmlb...