Solr
  1. Solr
  2. SOLR-4120

Collection API: Support for specifying a list of solrs to spread a new collection across

    Details

      Description

      When creating a new collection through the Collection API, the Overseer (handling the creation) will spread shards for this new collection across all live nodes.

      Sometimes you dont want a collection spread across all available nodes. Allow for the create operation of the Collection API, to take a createNodeSet parameter containing a list of Solr to spread the new shards across. If not provided it will just spread across all available nodes (default).

      For an example of a concrete case of usage see: https://issues.apache.org/jira/browse/SOLR-4114?focusedCommentId=13505506&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13505506

      1. SOLR-4120_trunk_r1421793.patch
        29 kB
        Per Steffensen
      2. SOLR-4120_trunk.patch
        10 kB
        Per Steffensen
      3. SOLR-4120.patch
        10 kB
        Per Steffensen

        Issue Links

          Activity

          Hide
          Per Steffensen added a comment - - edited

          SOLR-4120.patch

          Where does it fit
          • It fits on top of revision 1412602 of branch lucene_solr_4_0, where the patch for SOLR-4114 has already been applied. The following should work if you have a checkout of revision 1412602 of branch lucene_solr_4_0
            • cd <checkout-folder>
            • patch -s -p0 < SOLR-4114.patch
            • patch --ignore-whitespace -p0 < SOLR-4120.patch

          You need the --ignore-whitespace - at least with my version of patch on Show Leopard. Probably because I do not have the correct Solr code-style installed in my Eclipse. Hmmm, probably should do that.

          Content of the patch

          The patch modifies the create operation of the Solr Collection API, so that it allows to provide a list of Solrs that the shards for the new collection should be spread across

          • Param key: createNodeSet (OverseerCollectionProcessor.CREATE_NODE_SET)
          • Param value: comma-separated list of node-names (equal to the node-names received from ClusterState.getLiveNodes())
          • Param is not mandatory. If not provided the created collection will still have its shards spread across all live nodes
          Testing

          BasicDistributedZkTest.testCollectionAPI has been modified to also test this feature

          Show
          Per Steffensen added a comment - - edited SOLR-4120 .patch Where does it fit It fits on top of revision 1412602 of branch lucene_solr_4_0, where the patch for SOLR-4114 has already been applied. The following should work if you have a checkout of revision 1412602 of branch lucene_solr_4_0 cd <checkout-folder> patch -s -p0 < SOLR-4114 .patch patch --ignore-whitespace -p0 < SOLR-4120 .patch You need the --ignore-whitespace - at least with my version of patch on Show Leopard. Probably because I do not have the correct Solr code-style installed in my Eclipse. Hmmm, probably should do that. Content of the patch The patch modifies the create operation of the Solr Collection API, so that it allows to provide a list of Solrs that the shards for the new collection should be spread across Param key: createNodeSet (OverseerCollectionProcessor.CREATE_NODE_SET) Param value: comma-separated list of node-names (equal to the node-names received from ClusterState.getLiveNodes()) Param is not mandatory. If not provided the created collection will still have its shards spread across all live nodes Testing BasicDistributedZkTest.testCollectionAPI has been modified to also test this feature
          Hide
          Per Steffensen added a comment -

          SOLR-4120_trunk.patch

          Where does it fit
          • It fits trunk (5.x), where the patch for SOLR-4114 has already been applied. The following should work if you have a checkout of trunk
            • cd <checkout-folder>
            • patch -s -p0 < SOLR-4114_trunk.patch
            • patch -s -p0 < SOLR-4120_trunk.patch
          Content and tests

          Same as for the lucene_solr_4_0 patch

          Show
          Per Steffensen added a comment - SOLR-4120 _trunk.patch Where does it fit It fits trunk (5.x), where the patch for SOLR-4114 has already been applied. The following should work if you have a checkout of trunk cd <checkout-folder> patch -s -p0 < SOLR-4114 _trunk.patch patch -s -p0 < SOLR-4120 _trunk.patch Content and tests Same as for the lucene_solr_4_0 patch
          Hide
          Per Steffensen added a comment -

          Hope you will commit, and (as for SOLR-4114) consider backporting to 4.x, since we expect to upgrade to 4.1 when it is released, and we would really like this feature to be included.

          Show
          Per Steffensen added a comment - Hope you will commit, and (as for SOLR-4114 ) consider backporting to 4.x, since we expect to upgrade to 4.1 when it is released, and we would really like this feature to be included.
          Hide
          Mark Miller added a comment -

          Hey Per, since you probably still have this in a check out, might be faster for you to update it to trunk than me - unless I'm confused, all of the files have some small conflicts now.

          I can get to updating it if not, but if I can get you to do it, I will likely get to committing it faster.

          Show
          Mark Miller added a comment - Hey Per, since you probably still have this in a check out, might be faster for you to update it to trunk than me - unless I'm confused, all of the files have some small conflicts now. I can get to updating it if not, but if I can get you to do it, I will likely get to committing it faster.
          Hide
          Per Steffensen added a comment -

          I will find time soon to make you a trunk-based patch. Thanks, Mark!

          In the meantime you could look at SOLR-4140 I know I am pushing it now...

          Show
          Per Steffensen added a comment - I will find time soon to make you a trunk-based patch. Thanks, Mark! In the meantime you could look at SOLR-4140 I know I am pushing it now...
          Hide
          Mark Miller added a comment -

          Just a reminder ping - I'd like to see this get into 4.1.

          Show
          Mark Miller added a comment - Just a reminder ping - I'd like to see this get into 4.1.
          Hide
          Per Steffensen added a comment -

          I had a few hours today to make a new patch for trunk (that was what you asked for - not branch_4x). Expect you to backport it to branch_4x. New patch comming up.

          Show
          Per Steffensen added a comment - I had a few hours today to make a new patch for trunk (that was what you asked for - not branch_4x). Expect you to backport it to branch_4x. New patch comming up.
          Hide
          Mark Miller added a comment -

          trunk and 4x are pretty much the same right now - Ill take either.

          I also tend to just say trunk for up to date - stuck in the old days.

          Show
          Mark Miller added a comment - trunk and 4x are pretty much the same right now - Ill take either. I also tend to just say trunk for up to date - stuck in the old days.
          Hide
          Per Steffensen added a comment - - edited

          SOLR-4120_trunk_r1421793.patch

          Hurry up, get it in, Mark - while its still hot!
          Did not run entire test-suite after this patch, but I did run OverseerCollectionProcessorTest and BasicDistributedZkTest to verify that they are both green. Hope I did not ruin other tests - couldnt imagine.

          Where does it fit
          • It fits trunk (5.x) revision 1421793
          Content of patch
          • Same as for the lucene_solr_4_0 patch
          Testing
          • Same as for the lucene_solr_4_0 patch
          • Added tests for the feature in OverseerCollectionProcessorTest. Tests checking detailed combinations that does not belong in the "slow" tests in BasicDistributedZkTest. Also testing the SOLR-4114 and SOLR-4120 features in combination - that is, situations where createNodeSet is so small that the collection cannot be created living up to numShards and replicationFactor without violating the maxShardsPerNode (SOLR-4114) limit, but where it would have been possible to create the collection if either of maxShardsPerNode was higher or if createNodeSet had not been specified.
          Show
          Per Steffensen added a comment - - edited SOLR-4120 _trunk_r1421793.patch Hurry up, get it in, Mark - while its still hot! Did not run entire test-suite after this patch, but I did run OverseerCollectionProcessorTest and BasicDistributedZkTest to verify that they are both green. Hope I did not ruin other tests - couldnt imagine. Where does it fit It fits trunk (5.x) revision 1421793 Content of patch Same as for the lucene_solr_4_0 patch Testing Same as for the lucene_solr_4_0 patch Added tests for the feature in OverseerCollectionProcessorTest. Tests checking detailed combinations that does not belong in the "slow" tests in BasicDistributedZkTest. Also testing the SOLR-4114 and SOLR-4120 features in combination - that is, situations where createNodeSet is so small that the collection cannot be created living up to numShards and replicationFactor without violating the maxShardsPerNode ( SOLR-4114 ) limit, but where it would have been possible to create the collection if either of maxShardsPerNode was higher or if createNodeSet had not been specified.
          Hide
          Per Steffensen added a comment - - edited

          trunk for up to date

          Guess thats HEAD

          Show
          Per Steffensen added a comment - - edited trunk for up to date Guess thats HEAD
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1421999

          SOLR-4120: Collection API: Support for specifying a list of Solr addresses to spread a new collection across.

          Show
          Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1421999 SOLR-4120 : Collection API: Support for specifying a list of Solr addresses to spread a new collection across.
          Hide
          Mark Miller added a comment -

          Thanks Per!

          Show
          Mark Miller added a comment - Thanks Per!
          Hide
          Per Steffensen added a comment -

          No problem. You will add it to 4.x also, right?

          I'd like to see this get into 4.1

          Are you planing to release a 4.1 soon?

          Show
          Per Steffensen added a comment - No problem. You will add it to 4.x also, right? I'd like to see this get into 4.1 Are you planing to release a 4.1 soon?
          Hide
          Mark Miller added a comment -

          It's on 4.x - for some reason the commit bot has not dealt with it - I have to look into why.

          I'm hoping we release 4.1 very, very soon...

          Show
          Mark Miller added a comment - It's on 4.x - for some reason the commit bot has not dealt with it - I have to look into why. I'm hoping we release 4.1 very, very soon...
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1422018

          SOLR-4120: Collection API: Support for specifying a list of Solr addresses to spread a new collection across.

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1422018 SOLR-4120 : Collection API: Support for specifying a list of Solr addresses to spread a new collection across.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Per Steffensen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development