Solr
  1. Solr
  2. SOLR-5308

Split all documents of a route key into another collection

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 5.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Enable SolrCloud users to split out a set of documents from a source collection into another collection.

      This will be useful in multi-tenant environments. This feature will make it possible to split a tenant out of a collection and put them into their own collection which can be scaled separately.

      1. SOLR-5308-fixes.patch
        11 kB
        Shalin Shekhar Mangar
      2. SOLR-5308-bitsep-fix.patch
        15 kB
        Shalin Shekhar Mangar
      3. SOLR-5308.patch
        15 kB
        Shalin Shekhar Mangar
      4. SOLR-5308.patch
        17 kB
        Shalin Shekhar Mangar
      5. SOLR-5308.patch
        33 kB
        Shalin Shekhar Mangar
      6. SOLR-5308.patch
        33 kB
        Shalin Shekhar Mangar
      7. SOLR-5308.patch
        45 kB
        Shalin Shekhar Mangar
      8. SOLR-5308.patch
        50 kB
        Shalin Shekhar Mangar
      9. SOLR-5308.patch
        49 kB
        Shalin Shekhar Mangar
      10. SOLR-5308.patch
        56 kB
        Shalin Shekhar Mangar

        Issue Links

          Activity

          Hide
          Shalin Shekhar Mangar added a comment - - edited

          I think the way it can be made to work is by using the routing key (shard.key or as it is known now route) to split an index. The target collection must be specified as a request parameter. The target shard can be looked up automatically using the hash of the routing key against the shards of target collection. Optionally, requests for that shard.key can be automatically forwarded to the target collection for a limited amount of time once the split is complete.

          Show
          Shalin Shekhar Mangar added a comment - - edited I think the way it can be made to work is by using the routing key (shard.key or as it is known now route ) to split an index. The target collection must be specified as a request parameter. The target shard can be looked up automatically using the hash of the routing key against the shards of target collection. Optionally, requests for that shard.key can be automatically forwarded to the target collection for a limited amount of time once the split is complete.
          Hide
          Shalin Shekhar Mangar added a comment -

          Added a test to demonstrate that splitting by a specific route key is already possible at the SolrIndexSplitter level.

          Show
          Shalin Shekhar Mangar added a comment - Added a test to demonstrate that splitting by a specific route key is already possible at the SolrIndexSplitter level.
          Hide
          Shalin Shekhar Mangar added a comment -

          Assume that documents for a route key span 'N' shards in a collection 'collection1'. The same collection1 may also have data for other route keys as well. Suppose the route key 'A!' is receiving too much traffic and needs to migrated to another collection. The user shall create a new collection, say 'collection2' to hold data for key 'A!' and then call an API such as:

          /admin/collections?action=migrate&collection=collection1&split.key=A!&target.collection=collection2
          

          At this point, Solr determines the shards which overlap the hash range of the split key on both the source and target collections and for each such pair (shardX, shardY) invoke:

          /admin/collections?action=migrate&collection=collection1&split.key=A!&shard=shardX&target.collection=collection2&target.shard=shardY
          

          The above API will do the actual work of splitting and migrating index from source shard to target shard.

          Show
          Shalin Shekhar Mangar added a comment - Assume that documents for a route key span 'N' shards in a collection 'collection1'. The same collection1 may also have data for other route keys as well. Suppose the route key 'A!' is receiving too much traffic and needs to migrated to another collection. The user shall create a new collection, say 'collection2' to hold data for key 'A!' and then call an API such as: /admin/collections?action=migrate&collection=collection1&split.key=A!&target.collection=collection2 At this point, Solr determines the shards which overlap the hash range of the split key on both the source and target collections and for each such pair (shardX, shardY) invoke: /admin/collections?action=migrate&collection=collection1&split.key=A!&shard=shardX&target.collection=collection2&target.shard=shardY The above API will do the actual work of splitting and migrating index from source shard to target shard.
          Hide
          Shalin Shekhar Mangar added a comment -

          For splitting a single source shard into a single target collection/shard by a route key such as:

          /admin/collections?action=migrate&collection=collection1&split.key=A!&shard=shardX&target.collection=collection2&target.shard=shardY
          

          A rough strategy could be to:

          1. Create new core X on source
          2. Create new core Y on target
          3. Ask target core to buffer updates
          4. Start forwarding updates for route key received by source shard to target collection
          5. Split source shard to a new core X
          6. Ask Y to replicate fully from X
          7. Core Admin merge Y to target core
          8. Ask target core to replay buffer updates
          Show
          Shalin Shekhar Mangar added a comment - For splitting a single source shard into a single target collection/shard by a route key such as: /admin/collections?action=migrate&collection=collection1&split.key=A!&shard=shardX&target.collection=collection2&target.shard=shardY A rough strategy could be to: Create new core X on source Create new core Y on target Ask target core to buffer updates Start forwarding updates for route key received by source shard to target collection Split source shard to a new core X Ask Y to replicate fully from X Core Admin merge Y to target core Ask target core to replay buffer updates
          Hide
          Shalin Shekhar Mangar added a comment -

          This patch implements the MIGRATE Collection API

          1. Identifies the source and target shards for the given split.key. The for each pair of (source,target)
            1. Calculates common hash range by intersecting hash range of source shard, target shard and split.key's hash range
            2. Creates a temporary collection with just one replica on the source shard leader node
            3. Splits the source shard using the common hash range and split.key into the temporary collection
            4. Creates a replica of the temporary collection on the target shard leader's node
            5. Waits for it to become active
            6. Merges the replica's index into the target shard leader's index

          This is a rough cut to show where we're going with this. There are plenty of nocommits and todos. I still need to figure out how the request forwarding should work and if/how we can make the replica recovery asynchronous as we did for the split shard API.

          Show
          Shalin Shekhar Mangar added a comment - This patch implements the MIGRATE Collection API Identifies the source and target shards for the given split.key. The for each pair of (source,target) Calculates common hash range by intersecting hash range of source shard, target shard and split.key's hash range Creates a temporary collection with just one replica on the source shard leader node Splits the source shard using the common hash range and split.key into the temporary collection Creates a replica of the temporary collection on the target shard leader's node Waits for it to become active Merges the replica's index into the target shard leader's index This is a rough cut to show where we're going with this. There are plenty of nocommits and todos. I still need to figure out how the request forwarding should work and if/how we can make the replica recovery asynchronous as we did for the split shard API.
          Hide
          Shalin Shekhar Mangar added a comment -
          1. Added a REQUESTBUFFERUPDATES action in Core Admin API
          2. Use the above to ask target shard to buffer updates
          3. Call REQUESTAPPLYUPDATES action to replay buffered updates after index merge

          I'm still working through the nocommits.

          Show
          Shalin Shekhar Mangar added a comment - Added a REQUESTBUFFERUPDATES action in Core Admin API Use the above to ask target shard to buffer updates Call REQUESTAPPLYUPDATES action to replay buffered updates after index merge I'm still working through the nocommits.
          Hide
          Noble Paul added a comment -

          I feel that the REQUESTBUFFERUPDATES operation should not be a coreadmin action . It is doing an operation that is specific to the internals of a core. Can it not be an extra operation on the UpdateHandler?

          Show
          Noble Paul added a comment - I feel that the REQUESTBUFFERUPDATES operation should not be a coreadmin action . It is doing an operation that is specific to the internals of a core. Can it not be an extra operation on the UpdateHandler?
          Hide
          Shalin Shekhar Mangar added a comment -

          I feel that the REQUESTBUFFERUPDATES operation should not be a coreadmin action . It is doing an operation that is specific to the internals of a core. Can it not be an extra operation on the UpdateHandler?

          I think you're right. But this is just an implementation detail. It is by no means a public supported API. I had already added an operation to apply buffered updates for shard splitting so I added this in the same place. We can refactor this to be a update handler command. I'll open an issue.

          Show
          Shalin Shekhar Mangar added a comment - I feel that the REQUESTBUFFERUPDATES operation should not be a coreadmin action . It is doing an operation that is specific to the internals of a core. Can it not be an extra operation on the UpdateHandler? I think you're right. But this is just an implementation detail. It is by no means a public supported API. I had already added an operation to apply buffered updates for shard splitting so I added this in the same place. We can refactor this to be a update handler command. I'll open an issue.
          Hide
          Shalin Shekhar Mangar added a comment -
          1. Added a simple test that tests for migration of a route key from a static index (i.e. no updates in progress)
          2. Cleaning up existing temporary collections was broken in the last patch. I fixed it.

          I started working on forwarding requests in this patch. A RoutingRule is added to the clusterstate.json inside the source shard. Such a rule may be used by the DistributedUpdateProcessor to forward requests from source shard to target collection. Example below:

          "shard2":{
                        "range":"0-7fffffff",
                        "state":"active",
                        "replicas":{
                          "core_node2":{
                            "state":"active",
                            "base_url":"http://127.0.0.1:51321/ckvmo/du",
                            "core":"collection1",
                            "node_name":"127.0.0.1:51321_ckvmo%2Fdu",
                            "leader":"true"},
                          "core_node4":{
                            "state":"active",
                            "base_url":"http://127.0.0.1:51328/ckvmo/du",
                            "core":"collection1",
                            "node_name":"127.0.0.1:51328_ckvmo%2Fdu"}},
                        "routingRules":{"a!":[{
                              "targetShard":"shard1",
                              "routeRange":"3c250000-3c25ffff",
                              "expireAt":"1382449837040",
                              "targetCollection":"migrate_routekey_test_targetCollection",
                              "routeKey":"a!"}]}}}
          

          The expireAt is hard coded to System.currentTimeMillis() + 10 minutes but may be passed as a parameter in future. The intent is to forward update requests for the given route key and the given hash range from a source shard to the target collection (target shard doesn't matter) temporarily i.e. during the migration and for some time afterwards. This is still a work in progress.

          Show
          Shalin Shekhar Mangar added a comment - Added a simple test that tests for migration of a route key from a static index (i.e. no updates in progress) Cleaning up existing temporary collections was broken in the last patch. I fixed it. I started working on forwarding requests in this patch. A RoutingRule is added to the clusterstate.json inside the source shard. Such a rule may be used by the DistributedUpdateProcessor to forward requests from source shard to target collection. Example below: "shard2" :{ "range" : "0-7fffffff" , "state" : "active" , "replicas" :{ "core_node2" :{ "state" : "active" , "base_url" : "http: //127.0.0.1:51321/ckvmo/du" , "core" : "collection1" , "node_name" : "127.0.0.1:51321_ckvmo%2Fdu" , "leader" : " true " }, "core_node4" :{ "state" : "active" , "base_url" : "http: //127.0.0.1:51328/ckvmo/du" , "core" : "collection1" , "node_name" : "127.0.0.1:51328_ckvmo%2Fdu" }}, "routingRules" :{ "a!" :[{ "targetShard" : "shard1" , "routeRange" : "3c250000-3c25ffff" , "expireAt" : "1382449837040" , "targetCollection" : "migrate_routekey_test_targetCollection" , "routeKey" : "a!" }]}}} The expireAt is hard coded to System.currentTimeMillis() + 10 minutes but may be passed as a parameter in future. The intent is to forward update requests for the given route key and the given hash range from a source shard to the target collection (target shard doesn't matter) temporarily i.e. during the migration and for some time afterwards. This is still a work in progress.
          Hide
          Noble Paul added a comment - - edited

          is it possible that the same shard key spans multiple shards? in that case , will the same entry will be repeated in another shard also?

          why is this an array

          {"a!":[]}

          ? will there be multiple rules for the same shard key ?

          Show
          Noble Paul added a comment - - edited is it possible that the same shard key spans multiple shards? in that case , will the same entry will be repeated in another shard also? why is this an array {"a!":[]} ? will there be multiple rules for the same shard key ?
          Hide
          Shalin Shekhar Mangar added a comment -

          Is it possible that the same shard key spans multiple shards?

          Yes. By using the bitSeparator syntax for composite keys, a single shard key can span multiple shards.

          why is this an array

          Unknown macro: {"a!"}

          ? will there be multiple rules for the same shard key ?

          Yes there can be multiple rules for the same key because the ranges may span multiple target shards.

          Show
          Shalin Shekhar Mangar added a comment - Is it possible that the same shard key spans multiple shards? Yes. By using the bitSeparator syntax for composite keys, a single shard key can span multiple shards. why is this an array Unknown macro: {"a!"} ? will there be multiple rules for the same shard key ? Yes there can be multiple rules for the same key because the ranges may span multiple target shards.
          Hide
          Noble Paul added a comment - - edited

          We can have a simpler routing rule as follows

            "routingRules":{"a!":{
                              "ranges":["3c250000-3c25ffff"]
                              "expireAt":"1382449837040",
                              "targetCollection":"migrate_routekey_test_targetCollection"}}
          

          The target shard does not make sense because it is something that can be easily derived and may change if there is a further split on the target collection. note that the "ranges" is an array now so that multiple values can be set there

          Show
          Noble Paul added a comment - - edited We can have a simpler routing rule as follows "routingRules":{"a!":{ "ranges":["3c250000-3c25ffff"] "expireAt":"1382449837040", "targetCollection":"migrate_routekey_test_targetCollection"}} The target shard does not make sense because it is something that can be easily derived and may change if there is a further split on the target collection. note that the "ranges" is an array now so that multiple values can be set there
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Noble. That is certainly simpler. This patch has it as:

          "routingRules":{"a!":{
                            "routeRanges":"3c250000-3c25ffff",
                            "expireAt":"1382535453866",
                            "targetCollection":"migrate_routekey_test_targetCollection"}}
          

          I won't keep the routeRanges as a json list but as a comma-separated string.

          Show
          Shalin Shekhar Mangar added a comment - Thanks Noble. That is certainly simpler. This patch has it as: "routingRules" :{ "a!" :{ "routeRanges" : "3c250000-3c25ffff" , "expireAt" : "1382535453866" , "targetCollection" : "migrate_routekey_test_targetCollection" }} I won't keep the routeRanges as a json list but as a comma-separated string.
          Hide
          Shalin Shekhar Mangar added a comment -

          Changes:

          1. Added add request forwarding to target collection. The incoming request on a target collection is handled according to the state of the node. If the update log is in buffering mode then the request is buffered otherwise the version set by the source leader is stripped and leader logic is invoked.
          2. Added a test with request forwarding

          I'm still working on forwarding delete requests, remove routing rules after expiry and adding more/better tests.

          Show
          Shalin Shekhar Mangar added a comment - Changes: Added add request forwarding to target collection. The incoming request on a target collection is handled according to the state of the node. If the update log is in buffering mode then the request is buffered otherwise the version set by the source leader is stripped and leader logic is invoked. Added a test with request forwarding I'm still working on forwarding delete requests, remove routing rules after expiry and adding more/better tests.
          Hide
          Shalin Shekhar Mangar added a comment -

          This patch adds request forwarding for delete by ID and query requests.

          Show
          Shalin Shekhar Mangar added a comment - This patch adds request forwarding for delete by ID and query requests.
          Hide
          Shalin Shekhar Mangar added a comment -

          Removed unnecessary logging added to aid in debugging.

          Show
          Shalin Shekhar Mangar added a comment - Removed unnecessary logging added to aid in debugging.
          Hide
          Shalin Shekhar Mangar added a comment -
          1. Added routing rule expiry and test to assert behaviour.
          2. Added a forward.timeout parameter to specify the number of seconds after which a routing rule is expired. The default is 10 minutes.
          Show
          Shalin Shekhar Mangar added a comment - Added routing rule expiry and test to assert behaviour. Added a forward.timeout parameter to specify the number of seconds after which a routing rule is expired. The default is 10 minutes.
          Hide
          ASF subversion and git services added a comment -

          Commit 1541832 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1541832 ]

          SOLR-5308: A new 'migrate' collection API to split all documents with a route key into another collection

          Show
          ASF subversion and git services added a comment - Commit 1541832 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1541832 ] SOLR-5308 : A new 'migrate' collection API to split all documents with a route key into another collection
          Hide
          ASF subversion and git services added a comment -

          Commit 1542175 from shalin@apache.org in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1542175 ]

          SOLR-5308: A new 'migrate' collection API to split all documents with a route key into another collection

          Show
          ASF subversion and git services added a comment - Commit 1542175 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1542175 ] SOLR-5308 : A new 'migrate' collection API to split all documents with a route key into another collection
          Hide
          Shalin Shekhar Mangar added a comment -

          This patch fixes three bugs:

          1. No collection.configName was specified while creating the temporary collection. This caused it to fail when multiple config sets exist and the name of the config set is different from the collection name. I had to move the readConfigName method from ZkController to ZkStateReader because the OverseerCollectionProcessor does not have access to the ZkController.
          2. Fixed a NPE during the routing rule wait loop
          3. Fixed a race condition where the split is performed before the temp collection leader becomes active. In this patch we wait for the leader to become active before invoking split.
          Show
          Shalin Shekhar Mangar added a comment - This patch fixes three bugs: No collection.configName was specified while creating the temporary collection. This caused it to fail when multiple config sets exist and the name of the config set is different from the collection name. I had to move the readConfigName method from ZkController to ZkStateReader because the OverseerCollectionProcessor does not have access to the ZkController. Fixed a NPE during the routing rule wait loop Fixed a race condition where the split is performed before the temp collection leader becomes active. In this patch we wait for the leader to become active before invoking split.
          Hide
          ASF subversion and git services added a comment -

          Commit 1544414 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1544414 ]

          SOLR-5308: Use source collection's configName to create temp collection. Fixed NPE routing rule wait loop. Wait for temp collection leader to be active before splitting the source index.

          Show
          ASF subversion and git services added a comment - Commit 1544414 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1544414 ] SOLR-5308 : Use source collection's configName to create temp collection. Fixed NPE routing rule wait loop. Wait for temp collection leader to be active before splitting the source index.
          Hide
          ASF subversion and git services added a comment -

          Commit 1544415 from shalin@apache.org in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1544415 ]

          SOLR-5308: Use source collection's configName to create temp collection. Fixed NPE routing rule wait loop. Wait for temp collection leader to be active before splitting the source index.

          Show
          ASF subversion and git services added a comment - Commit 1544415 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1544415 ] SOLR-5308 : Use source collection's configName to create temp collection. Fixed NPE routing rule wait loop. Wait for temp collection leader to be active before splitting the source index.
          Hide
          Shalin Shekhar Mangar added a comment -

          This patch fixes two bugs:

          1. If migrate was invoked with a split.key containing a bit separator then the forwarding logic didn't kick in
          2. When forwarding request to target collection, an intermediate forwarding node would cause the distrib.from.shard and distrib.from.collection params to be removed and the request would be rejected by the target shard. Now requests are routed directly to target shard instead of random assignment.
          Show
          Shalin Shekhar Mangar added a comment - This patch fixes two bugs: If migrate was invoked with a split.key containing a bit separator then the forwarding logic didn't kick in When forwarding request to target collection, an intermediate forwarding node would cause the distrib.from.shard and distrib.from.collection params to be removed and the request would be rejected by the target shard. Now requests are routed directly to target shard instead of random assignment.
          Hide
          ASF subversion and git services added a comment -

          Commit 1549821 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1549821 ]

          SOLR-5308: Handle route keys with bit separators. Route docs to target shard leader directly.

          Show
          ASF subversion and git services added a comment - Commit 1549821 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1549821 ] SOLR-5308 : Handle route keys with bit separators. Route docs to target shard leader directly.
          Hide
          ASF subversion and git services added a comment -

          Commit 1549822 from shalin@apache.org in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1549822 ]

          SOLR-5308: Handle route keys with bit separators. Route docs to target shard leader directly.

          Show
          ASF subversion and git services added a comment - Commit 1549822 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1549822 ] SOLR-5308 : Handle route keys with bit separators. Route docs to target shard leader directly.

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development