[LUCENE-4550] For extremely wide shapes (> 180 degrees) distErrPct is not used correctly

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0
• Fix Version/s: 4.2, 6.0
• Component/s: modules/spatial
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 09/Nov/12 16:08
• Updated: 28/Aug/22 13:32
• Resolved: 22/Jan/13 21:26

Description

When a shape is given to a PrefixTreeStrategy (index or query time), it needs to know how many levels down the prefix tree to go for a target precision (distErrPct). distErrPct is basically a fraction of the radius of the shape, defaulting to 2.5% (0.0025).

If the shape presented is extremely wide, > 180 degrees, then the internal calculations in SpatialArgs.calcDistanceFromErrPct(...) will wrongly measure the shape's size as having width < 180 degrees, yielding more accuracy than intended. Given that this happens for unrealistic shape sizes and results in more accuracy, I am flagging this as "minor", but a bug nonetheless. Indeed, this was discovered as a result of someone using lucene-spatial incorrectly, not for an actual shape they have. But in the extreme [erroneous] case they had, they had 566k terms generated, when it should have been ~1k tops.