Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-15109

Optimize shard splitByPrefix logic to reduce number of splits required

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      The goal of SplitByPrefix logic is to identify "buckets" within a shard that contain documents that should be co-located (according to their doc prefix), and split such that those buckets are preserved. One issue that we have found with splitByPrefix in practice is that it often takes several splits to isolate a particularly large bucket within the hash range.

      David Smiley came up with a simple optimization that will reduce the number of splits needed to isolate such a bucket:

      Loop over all RangeCounts... does it intersect the middle third of the input? If not, move on. If so, track the biggest. When this loop finishes, you will have the biggest that also intersects the middle third. Then simply choose the side of this biggest RangeCount that is closest to the middle of the input range.

      This should be clearer with the following diagrams:

        Attachments

        1. Split 3 (1).png
          13 kB
          Megan Carey
        2. Split 2 (1).png
          17 kB
          Megan Carey
        3. Split 1 (1).png
          19 kB
          Megan Carey

          Activity

            People

            • Assignee:
              dsmiley David Smiley
              Reporter:
              megancarey Megan Carey
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: