[LUCENE-8562] Speed up merging segments of points with data dimensions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.7, 8.0
• Fix Version/s: 7.7, 8.0
• Component/s: core/index
• Lucene Fields: New
• Assignee: Ignacio Vera
• Reporter: Ignacio Vera
• Created: 11/Nov/18 11:59
• Updated: 28/Aug/22 15:37
• Resolved: 27/Nov/18 09:30

Description

Currently when merging segments of points with data dimensions, all dimensions are sorted and carried over down the tree even though only indexing dimensions are needed to build the BKD tree. This is needed so leaf node data can be compressed by common prefix.

But when using MutablePointValues, this ordering is done at the leaf level so we can se a similar approach from data dimensions and delay the sorting at leaf level. This seems to speed up indexing time as well as reduce the storage needed for building the index.