[LUCENE-4698] Overhaul ShapeFieldCache because its a memory pig

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: modules/spatial
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: David Smiley
• Created: 20/Jan/13 21:44
• Updated: 28/Aug/22 13:36
• Resolved: 12/Mar/17 04:26

Description

The org.apache.lucene.spatial.util.ShapeFieldCache* classes together implement a spatial field cache for points, similar to FieldCache for other fields. It supports a variable number of points per document, and it's currently only used by the SpatialPrefixTree strategy because that's the only strategy that supports a variable number of points per document. The other spatial strategies use the FieldCache. The ShapeFieldCache has problems:

• It's a memory pig. Each point is stored as a Point object, instead of an array of x & y coordinates. Furthermore, each Point is in an ArrayList that exists for each Document. It's not done any differently when your spatial data isn't multi-valued.
• The cache is not per-segment, it's per-IndexReader, thereby making it un-friendly to NRT search.
• The cache entries don't self-expire optimally to free up memory. The cache is simply stored in a WeakHashMap<IndexReader,ShapeFieldCache>. The big cache entries are only freed when the WeakHashMap is used and the JVM realizes the IndexSearcher instance has been GC'ed.

Issue Links

• is depended upon by: LUCENE-4869 Optimize IsWithin spatial RPT to use a point cache for false-positive removal
• relates to: SOLR-4329 Have DocumentBuilder give value collections to the FieldType
• relates to: SOLR-5170 Spatial multi-value distance sort via DocValues