(In reply to comment #11)
> andreas, any proposed resolution for this?
Two proposals were already mentioned, admittedly rather vague. All that's left is the decision, which should be pretty simple. I would likely have committed the fix and closed the issue last year, if others had not insisted that FOP needs to avoid rounding altogether...
There is the so-called 'ideal' or 'proper' solution, which would be a more long-term effort that involves a thorough impact analysis. A switch to values in whatever unit, stored internally as float or double, will have consequences all over the codebase, and for what?
Just for the sake of theoretical, mathematical precision -
beyond the third decimal, for a value expressed in pt...? Please! :/
This would cost quite some time and effort for a prize that is hardly worth it in this context.
I do not see why FOP would even /need/ that level of precision, given the scale. It's not like FOP is in the business of splitting atoms, so settling on 1/1000pt, internally stored as integers is really not that bad. We are only talking about a few tenths of a µm of difference due to loss in precision, here. Oh well, maybe it's just me...
A more 'appropriate' solution, at least from a practical, cost/benefit perspective, would be to simply allow for a margin in the particular comparison triggering the reported issue.
The line of code in question in PageBreakingAlgorithm[*] could be made to consider a value of 510237(mpt) as 'equal' to 510236(mpt).
Sounds reasonable, keeping in mind that:
1° 'mpt' is an unofficial unit anyway, so FOP determines the calculation rules; why not have "1=2" when counting in those units?
2° converted back to standard units, the loss of 0.001pt would only yield a discernible difference provided that the output resolution is --yep, 72 thousand dpi.
The latter proposal is a single-line fix for this particular bug entry. The attachment could then basically serve as the only test case, extended with a few other cases, trying out different combinations of attributes/unit specs that yield the largest conceivable rounding differences when run through FixedLength.convert().
[*] line 1138: 'Math.abs (...) <= m' instead of '... != 0', where m is half the amount of margin, in 'mpt', within which two page-widths are considered identical.
As a suggestion, I mentioned 0.005pt earlier, or around 0.2µm. Interestingly, I found out later that typical light microscopes, assuming visible range light, have a theoretical resolution limit of just about that value. Might count as an argument pro: if your output target supported such a high pixel density, you would probably even miss it /even/ if you looked at it through a conventional microscope.