[LUCENE-9149] Increase data dimension limit in BKD

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Nick Knize
• Created: 17/Jan/20 21:09
• Updated: 28/Aug/22 15:56
• Resolved: 10/Feb/20 18:39

Description

LUCENE-8496 added selective indexing; the ability to designate the first K <= N dimensions for driving the construction of the BKD internal nodes. Follow on work stored the "data dimensions" for only the leaf nodes and only the "index dimensions" are stored for the internal nodes. While maxPointsInLeafNode is still important for managing the BKD heap memory footprint (thus we don't want this to get too large), I'd like to propose increasing the MAX_DIMENSIONS limit (to something not too crazy like 16; effectively doubling the index dimension limit) while maintaining the MAX_INDEX_DIMENSIONS at 8.

Doing this will enable us to encode higher dimension data within a lower dimension index (e.g., 3D tessellated triangles as a 10 dimension point using only the first 6 dimensions for index construction)

 

Issue Links

• links to: GitHub Pull Request #1182