Solr
  1. Solr
  2. SOLR-5300

Split shards with custom hash ranges

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.6, Trunk
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Currently, shards can only be split at the mid point of their hash range. This makes it difficult to control the distribution of data in the sub shards.

      We should make it possible to specify ranges to be used for splitting. A ranges parameter can be added which can accept hash ranges in hexadecimal e.g. ranges=0-1f4,1f5-3e8,3e9-5dc will split a shard with range 0-1500 into three shards with ranges [0,500], [501-1000] and [1001-1500] respectively.

      1. SOLR-5300.patch
        12 kB
        Shalin Shekhar Mangar
      2. SOLR-5300.patch
        11 kB
        Shalin Shekhar Mangar
      3. SOLR-5300-cover-shardrange.patch
        5 kB
        Shalin Shekhar Mangar

        Activity

        Hide
        Shalin Shekhar Mangar added a comment -

        A new parameter "ranges" is introduced in split shard action which accepts comma-separated hash ranges in hexadecimal only.

        Show
        Shalin Shekhar Mangar added a comment - A new parameter "ranges" is introduced in split shard action which accepts comma-separated hash ranges in hexadecimal only.
        Hide
        Shalin Shekhar Mangar added a comment -

        Added a check to make sure that specified hash range is a subset of parent hash range.

        Show
        Shalin Shekhar Mangar added a comment - Added a check to make sure that specified hash range is a subset of parent hash range.
        Hide
        ASF subversion and git services added a comment -

        Commit 1529444 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1529444 ]

        SOLR-5300: Shards can be split by specifying arbitrary number of hash ranges within the shard's hash range

        Show
        ASF subversion and git services added a comment - Commit 1529444 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1529444 ] SOLR-5300 : Shards can be split by specifying arbitrary number of hash ranges within the shard's hash range
        Hide
        ASF subversion and git services added a comment -

        Commit 1529445 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1529445 ]

        SOLR-5300: Shards can be split by specifying arbitrary number of hash ranges within the shard's hash range

        Show
        ASF subversion and git services added a comment - Commit 1529445 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1529445 ] SOLR-5300 : Shards can be split by specifying arbitrary number of hash ranges within the shard's hash range
        Hide
        Jessica Cheng Mallet added a comment -

        I see that the current code sanity checks that each supplied sub-ranges is in fact a sub-range of the original range--should we also check that the supplied sub-ranges actually covers the entire range of the shard?

        Show
        Jessica Cheng Mallet added a comment - I see that the current code sanity checks that each supplied sub-ranges is in fact a sub-range of the original range--should we also check that the supplied sub-ranges actually covers the entire range of the shard?
        Hide
        Shalin Shekhar Mangar added a comment -

        should we also check that the supplied sub-ranges actually covers the entire range of the shard?

        Yes, you are right. I'll fix.

        Show
        Shalin Shekhar Mangar added a comment - should we also check that the supplied sub-ranges actually covers the entire range of the shard? Yes, you are right. I'll fix.
        Hide
        Shalin Shekhar Mangar added a comment -
        1. DocRouter.Range implements Comparable
        2. Split shard copies the ranges provided, sorts them and checks that they cover the entire hash range
        3. Added a test in ShardSplitTest
        Show
        Shalin Shekhar Mangar added a comment - DocRouter.Range implements Comparable Split shard copies the ranges provided, sorts them and checks that they cover the entire hash range Added a test in ShardSplitTest
        Hide
        ASF subversion and git services added a comment -

        Commit 1534974 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1534974 ]

        SOLR-5300: Check that the supplied hash ranges actually cover the entire range of the shard

        Show
        ASF subversion and git services added a comment - Commit 1534974 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1534974 ] SOLR-5300 : Check that the supplied hash ranges actually cover the entire range of the shard
        Hide
        ASF subversion and git services added a comment -

        Commit 1534975 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1534975 ]

        SOLR-5300: Check that the supplied hash ranges actually cover the entire range of the shard

        Show
        ASF subversion and git services added a comment - Commit 1534975 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1534975 ] SOLR-5300 : Check that the supplied hash ranges actually cover the entire range of the shard

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Shalin Shekhar Mangar
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development