[SOLR-15109] Optimize shard splitByPrefix logic to reduce number of splits required

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SolrCloud
• Assignee: David Smiley
• Reporter: Megan Carey
• Created: 26/Jan/21 20:05
• Updated: 13/Feb/21 00:08

Description

The goal of SplitByPrefix logic is to identify "buckets" within a shard that contain documents that should be co-located (according to their doc prefix), and split such that those buckets are preserved. One issue that we have found with splitByPrefix in practice is that it often takes several splits to isolate a particularly large bucket within the hash range.

dsmiley came up with a simple optimization that will reduce the number of splits needed to isolate such a bucket:

Loop over all RangeCounts... does it intersect the middle third of the input? If not, move on. If so, track the biggest. When this loop finishes, you will have the biggest that also intersects the middle third. Then simply choose the side of this biggest RangeCount that is closest to the middle of the input range.

This should be clearer with the following diagrams: