|
The solution provided works upto 4 decimal places.
That is 77.9999 is converted to sring properly. However 77.99999 becomes 78. Lets use %g18 for double and %g9 for float in sprintf for serializing.
This means we have reduced error rate as far as precision is concerned. BTW XSD spec says: decimal represents a subset of the real numbers, which can be represented by decimal numerals. The - value space- of decimal is the set of numbers that can be obtained by multiplying an integer by a non-positive power of ten, i.e., expressible as i × 10^-n where i and n are integers and n >= 0. Precision is not reflected in this value space; the number 2.0 is not distinct from the number 2.00. The - order-relation- on decimal is the order relation on real numbers, restricted to this subset. float is patterned after the IEEE single-precision 32-bit floating point type [IEEE 754-1985]. The basic - value space- of float consists of the values m × 2^e, where m is an integer whose absolute value is less than 2^24, and e is an integer between -149 and 104, inclusive. In addition to the basic - value space- described above, the - value space- of float also contains the following three special values: positive and negative infinity and not-a-number (NaN). The - order-relation- on float is: x < y iff y - x is positive for x and y in the value space. Positive infinity is greater than all other non-NaN values. NaN equals itself but is - incomparable- with (neither greater than nor less than) any other value in the - value space- . The double datatype is patterned after the IEEE double-precision 64-bit floating point type [IEEE 754-1985]. The basic - value space- of double consists of the values m × 2^e, where m is an integer whose absolute value is less than 2^53, and e is an integer between -1075 and 970, inclusive. In addition to the basic - value space- described above, the - value space- of double also contains the following three special values: positive and negative infinity and not-a-number (NaN). The - order-relation- on double is: x < y iff y - x is positive for x and y in the value space. Positive infinity is greater than all other non-NaN values. NaN equals itself but is - incomparable- with (neither greater than nor less than) any other value in the - value space- . I tried with %g9 (for float) and %g18 (for decimal) and there still seems to be more problems. The precision we get really depend on the number of digits we have in the value we try to deserialize.
We could only reach some level of precision but not full level The problem is somewhat solved with the use of %g. The user would need to decide the level of precision they need and adust this in the engine if precision is of a real concern.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Major
float number, 77.99 becomes 77.989998 in output string
AxisSprintf (m_Buf, BTS_BUFFSIZE, "%f", *((float*)(pValue)));
to
AxisSprintf (m_Buf, BTS_BUFFSIZE, "%g", *((float*)(pValue)));
According to the man page of sprintf %f causes the argument to be
"rounded and converted", %g does not.