Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-15109

Optimize shard splitByPrefix logic to reduce number of splits required

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • SolrCloud
    • None

    Description

      The goal of SplitByPrefix logic is to identify "buckets" within a shard that contain documents that should be co-located (according to their doc prefix), and split such that those buckets are preserved. One issue that we have found with splitByPrefix in practice is that it often takes several splits to isolate a particularly large bucket within the hash range.

      dsmiley came up with a simple optimization that will reduce the number of splits needed to isolate such a bucket:

      Loop over all RangeCounts... does it intersect the middle third of the input? If not, move on. If so, track the biggest. When this loop finishes, you will have the biggest that also intersects the middle third. Then simply choose the side of this biggest RangeCount that is closest to the middle of the input range.

      This should be clearer with the following diagrams:

      Attachments

        1. Split 3 (1).png
          13 kB
          Megan Carey
        2. Split 2 (1).png
          17 kB
          Megan Carey
        3. Split 1 (1).png
          19 kB
          Megan Carey

        Activity

          People

            dsmiley David Smiley
            megancarey Megan Carey
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: