Solr
  1. Solr
  2. SOLR-7766

support creation of a coreless collection

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      By supplying the special value of EMPTY for the createNodeSet parameter (via .../solr/admin/collections?action=CREATE&createNodeSet=EMPTY&name=myCollection&...) a collection can be created in the usual manner but it will just not yet contain any cores for any of its shards. Later on .../solr/admin/collections?wt=json&action=ADDREPLICA&collection=myCollection&... calls can create cores as and when and where required.

      github pull request with proposed changes to follow.

      1. SOLR-7766.patch
        15 kB
        Ramkumar Aiyengar
      2. SOLR-7766.txt
        18 kB
        Christine Poerschke

        Issue Links

          Activity

          Hide
          ASF GitHub Bot added a comment -

          GitHub user cpoerschke opened a pull request:

          https://github.com/apache/lucene-solr/pull/186

          SOLR-7766: support creation of a coreless collection

          for https://issues.apache.org/jira/i#browse/SOLR-7766

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/bloomberg/lucene-solr trunk-create-coreless-collection

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/186.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #186


          commit 74d2d93b24dcb4e8aa4f72a386b2a1f02abd7aa9
          Author: Christine Poerschke <cpoerschke@bloomberg.net>
          Date: 2015-04-02T16:22:04Z

          SOLR-????: support creation of a coreless collection

          By supplying a deliberately empty create node set (via .../solr/admin/collections?action=CREATE&createNodeSet=&name=myCollection&...) the collection can be created in the usual manner but it will just not yet contain any cores for any of its shards. Later on .../solr/admin/collections?wt=json&action=ADDREPLICA&collection=myCollection&... calls can create cores as and when and where required.

          The change to suppport this small new feature is in OverseerCollectionProcessor.java and in TestMiniSolrCloudCluster a new test case (testCollectionCreateWithoutCoresThenDelete) was added.


          Show
          ASF GitHub Bot added a comment - GitHub user cpoerschke opened a pull request: https://github.com/apache/lucene-solr/pull/186 SOLR-7766 : support creation of a coreless collection for https://issues.apache.org/jira/i#browse/SOLR-7766 You can merge this pull request into a Git repository by running: $ git pull https://github.com/bloomberg/lucene-solr trunk-create-coreless-collection Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #186 commit 74d2d93b24dcb4e8aa4f72a386b2a1f02abd7aa9 Author: Christine Poerschke <cpoerschke@bloomberg.net> Date: 2015-04-02T16:22:04Z SOLR-????: support creation of a coreless collection By supplying a deliberately empty create node set (via .../solr/admin/collections?action=CREATE&createNodeSet=&name=myCollection&...) the collection can be created in the usual manner but it will just not yet contain any cores for any of its shards. Later on .../solr/admin/collections?wt=json&action=ADDREPLICA&collection=myCollection&... calls can create cores as and when and where required. The change to suppport this small new feature is in OverseerCollectionProcessor.java and in TestMiniSolrCloudCluster a new test case (testCollectionCreateWithoutCoresThenDelete) was added.
          Hide
          Ramkumar Aiyengar added a comment -

          Attaching a updated patch with conflicts resolved in trunk..

          Show
          Ramkumar Aiyengar added a comment - Attaching a updated patch with conflicts resolved in trunk..
          Hide
          Christine Poerschke added a comment -

          Rebasing patch against latest trunk (to include SOLR-7823 changes), existing patch fails async=... tests.

          Show
          Christine Poerschke added a comment - Rebasing patch against latest trunk (to include SOLR-7823 changes), existing patch fails async=... tests.
          Hide
          Shalin Shekhar Mangar added a comment -

          How about replicationFactor=0 instead of an empty createNodeSet parameter?

          Show
          Shalin Shekhar Mangar added a comment - How about replicationFactor=0 instead of an empty createNodeSet parameter?
          Hide
          Mark Miller added a comment -

          replicationFactor can also be used as a target for things like autoAddReplicas. I think the empty createNodeSet makes a lot more sense.

          Show
          Mark Miller added a comment - replicationFactor can also be used as a target for things like autoAddReplicas. I think the empty createNodeSet makes a lot more sense.
          Hide
          Shalin Shekhar Mangar added a comment -

          replicationFactor can also be used as a target for things like autoAddReplicas. I think the empty createNodeSet makes a lot more sense.

          You forget that we now have a modify collection API which lets you change the replication factor among other things. Specifying an empty parameter to change the behavior of an API seems very hacky to me. An empty parameter should either have no effect or it should result in an error.

          Show
          Shalin Shekhar Mangar added a comment - replicationFactor can also be used as a target for things like autoAddReplicas. I think the empty createNodeSet makes a lot more sense. You forget that we now have a modify collection API which lets you change the replication factor among other things. Specifying an empty parameter to change the behavior of an API seems very hacky to me. An empty parameter should either have no effect or it should result in an error.
          Hide
          Mark Miller added a comment - - edited

          No, I know you can change it.

          It simply doesn't make sense to me like I said. You are not asking for a target replication factor of 0. You are asking that no nodes get a core. Having an empty createNodeSet matches the intent much more than messing with replication factor.

          Show
          Mark Miller added a comment - - edited No, I know you can change it. It simply doesn't make sense to me like I said. You are not asking for a target replication factor of 0. You are asking that no nodes get a core. Having an empty createNodeSet matches the intent much more than messing with replication factor.
          Hide
          Mark Miller added a comment - - edited

          API seems very hacky to me.

          What's the hack exactly?

          • If the issue is really just an empty param (something I don't agree is a hack), you could simply have a special keyword of createNodeSet=none.
          Show
          Mark Miller added a comment - - edited API seems very hacky to me. What's the hack exactly? If the issue is really just an empty param (something I don't agree is a hack), you could simply have a special keyword of createNodeSet=none.
          Hide
          Christine Poerschke added a comment -

          I agree re: the differentiation between createNodeSet missing and createNodeSet= present-and-empty is very subtle. Would a special value e.g. createNodeSet=EMPTY be clearer?

          The way i read OverseerCollectionProcessor.java line 2233 currently an empty parameter would result in an error, right? Draft Collections API docs list 'empty' as the default though (should it be 'blank box' as for 'async'?).

          Will update github pull request against latest trunk (with createNodeSet=) shortly and definitely hold off on any commits until agreement on how to request coreless collection creation is reached.

          Show
          Christine Poerschke added a comment - I agree re: the differentiation between createNodeSet missing and createNodeSet= present-and-empty is very subtle. Would a special value e.g. createNodeSet=EMPTY be clearer? The way i read OverseerCollectionProcessor.java line 2233 currently an empty parameter would result in an error, right? Draft Collections API docs list 'empty' as the default though (should it be 'blank box' as for 'async'?). Will update github pull request against latest trunk (with createNodeSet=) shortly and definitely hold off on any commits until agreement on how to request coreless collection creation is reached.
          Hide
          Mark Miller added a comment -

          createNodeSet=EMPTY be clearer?

          lol - cross post with my edit. I would prefer that to abusing replicationFactor.

          Show
          Mark Miller added a comment - createNodeSet=EMPTY be clearer? lol - cross post with my edit. I would prefer that to abusing replicationFactor.
          Hide
          Shalin Shekhar Mangar added a comment -

          Would a special value e.g. createNodeSet=EMPTY be clearer?

          Yeah, createNodeSet=EMPTY or createNodeSet=NONE would be much clearer than an empty parameter.

          Show
          Shalin Shekhar Mangar added a comment - Would a special value e.g. createNodeSet=EMPTY be clearer? Yeah, createNodeSet=EMPTY or createNodeSet=NONE would be much clearer than an empty parameter.
          Hide
          Mark Miller added a comment -

          Consensus reached.

          Show
          Mark Miller added a comment - Consensus reached.
          Hide
          Christine Poerschke added a comment - - edited

          Alrighty, let's go for createNodeSet=someSpecialValue - i will have a look around (on Monday) to see if 'empty' vs. 'EMPTY' vs. 'NONE' there is a precedent already, so that for consistency the same could be used here? Thanks Shalin Shekhar Mangar and Mark Miller for your input!

          PS: github pull request now updated (still has createNodeSet= though, will be updating both code and the newly added test case to use the special value).

          Show
          Christine Poerschke added a comment - - edited Alrighty, let's go for createNodeSet=someSpecialValue - i will have a look around (on Monday) to see if 'empty' vs. 'EMPTY' vs. 'NONE' there is a precedent already, so that for consistency the same could be used here? Thanks Shalin Shekhar Mangar and Mark Miller for your input! PS: github pull request now updated (still has createNodeSet= though, will be updating both code and the newly added test case to use the special value).
          Hide
          Mark Miller added a comment -

          I think EMPTY is probably the right choice - I was just shooting from the hip with none.

          Show
          Mark Miller added a comment - I think EMPTY is probably the right choice - I was just shooting from the hip with none.
          Hide
          Christine Poerschke added a comment -

          createNodeSet=EMPTY it is then. github pull request updated (also rebased against latest trunk) and also attached patch to this JIRA.

          Show
          Christine Poerschke added a comment - createNodeSet=EMPTY it is then. github pull request updated (also rebased against latest trunk) and also attached patch to this JIRA.
          Hide
          Christine Poerschke added a comment -

          Hello. If there are no further comments, suggestions or objections, i will go ahead and commit the latest patch/github pull request tomorrow (Wednesday), hopefully still in time for the 5.3 release. Or if near-missing it then would this be a candidate for merging back after the lucene_solr_5_3 branch is created?

          Show
          Christine Poerschke added a comment - Hello. If there are no further comments, suggestions or objections, i will go ahead and commit the latest patch/github pull request tomorrow (Wednesday), hopefully still in time for the 5.3 release. Or if near-missing it then would this be a candidate for merging back after the lucene_solr_5_3 branch is created?
          Hide
          Shalin Shekhar Mangar added a comment -

          +1 to commit

          Show
          Shalin Shekhar Mangar added a comment - +1 to commit
          Hide
          ASF subversion and git services added a comment -

          Commit 1694181 from Christine Poerschke in branch 'dev/trunk'
          [ https://svn.apache.org/r1694181 ]

          SOLR-7766: support creation of a coreless collection via createNodeSet=EMPTY

          Show
          ASF subversion and git services added a comment - Commit 1694181 from Christine Poerschke in branch 'dev/trunk' [ https://svn.apache.org/r1694181 ] SOLR-7766 : support creation of a coreless collection via createNodeSet=EMPTY
          Hide
          Christine Poerschke added a comment -

          ran 'svn merge' from trunk to branch_5x in the usual manner, no conflicts encountered but 'ant test' fails - checking

          Show
          Christine Poerschke added a comment - ran 'svn merge' from trunk to branch_5x in the usual manner, no conflicts encountered but 'ant test' fails - checking
          Hide
          Christine Poerschke added a comment -

          seeing this in the test failure:

             [junit4]   2> 62928 ERROR (TEST-TestAuthenticationFramework.testBasics-seed#[F8DED25E22E9AF52]) [    ] o.a.s.c.s.i.CloudSolrClient Request to collection testAuthenticationFrameworkCollection failed due to (510) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:64008/solr/testAuthenticationFrameworkCollection: Expected mime type application/octet-stream but got text/html. <html>
             [junit4]   2> <head>
             [junit4]   2> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
             [junit4]   2> <title>Error 510 </title>
             [junit4]   2> </head>
             [junit4]   2> <body>
             [junit4]   2> <h2>HTTP ERROR: 510</h2>
             [junit4]   2> <p>Problem accessing /solr/testAuthenticationFrameworkCollection/select. Reason:
             [junit4]   2> <pre>    {msg={&quot;testAuthenticationFrameworkCollection&quot;:15},code=510}</pre></p>
             [junit4]   2> <hr /><i><small>Powered by Jetty://</small></i>
             [junit4]   2> </body>
             [junit4]   2> </html>
          

          ran 'ant test -Dtestcase=TestAuthenticationFramework' ten times and half the time got that error. will investigate further before committing to branch_5x

          Show
          Christine Poerschke added a comment - seeing this in the test failure: [junit4] 2> 62928 ERROR (TEST-TestAuthenticationFramework.testBasics-seed#[F8DED25E22E9AF52]) [ ] o.a.s.c.s.i.CloudSolrClient Request to collection testAuthenticationFrameworkCollection failed due to (510) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http: //127.0.0.1:64008/solr/testAuthenticationFrameworkCollection: Expected mime type application/octet-stream but got text/html. <html> [junit4] 2> <head> [junit4] 2> <meta http-equiv= "Content-Type" content= "text/html;charset=ISO-8859-1" /> [junit4] 2> <title>Error 510 </title> [junit4] 2> </head> [junit4] 2> <body> [junit4] 2> <h2>HTTP ERROR: 510</h2> [junit4] 2> <p>Problem accessing /solr/testAuthenticationFrameworkCollection/select. Reason: [junit4] 2> <pre> {msg={&quot;testAuthenticationFrameworkCollection&quot;:15},code=510}</pre></p> [junit4] 2> <hr /><i><small>Powered by Jetty: //</small></i> [junit4] 2> </body> [junit4] 2> </html> ran 'ant test -Dtestcase=TestAuthenticationFramework' ten times and half the time got that error. will investigate further before committing to branch_5x
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          I am seeing the 510 error locally on trunk, but the test is passing anyway.

          At trunk, Jenkins reports a 401 error:
          http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2586/testReport/junit/org.apache.solr.cloud/TestAuthenticationFramework/testCollectionCreateWithoutCoresThenDelete/

          At first glance, I can't see anything obvious that would cause this 401.

          Show
          Ishan Chattopadhyaya added a comment - - edited I am seeing the 510 error locally on trunk, but the test is passing anyway. At trunk, Jenkins reports a 401 error: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2586/testReport/junit/org.apache.solr.cloud/TestAuthenticationFramework/testCollectionCreateWithoutCoresThenDelete/ At first glance, I can't see anything obvious that would cause this 401.
          Hide
          Christine Poerschke added a comment -

          I will try and see if the SOLR-6971 dumper patch helps here. Perhaps creating a coreless collection is quicker in some just-so way to make it more likely to cause the error?

          Show
          Christine Poerschke added a comment - I will try and see if the SOLR-6971 dumper patch helps here. Perhaps creating a coreless collection is quicker in some just-so way to make it more likely to cause the error?
          Hide
          Christine Poerschke added a comment -

          Nothing interesting from SOLR-6971 dumper patch so far, but have a theory re: TestAuthenticationFramework test method call ordering, will add further notes on SOLR-7877.

          Show
          Christine Poerschke added a comment - Nothing interesting from SOLR-6971 dumper patch so far, but have a theory re: TestAuthenticationFramework test method call ordering, will add further notes on SOLR-7877 .
          Hide
          Christine Poerschke added a comment -

          Working on merging this from trunk to branch_5x - some minor merge conflicts need resolving.

          also still to do: merge to lucene_solr_5_3 or move this issue out of the 5.3 CHANGES.txt section (Shalin Shekhar Mangar just created the 5.4 section - thank you!)

          Show
          Christine Poerschke added a comment - Working on merging this from trunk to branch_5x - some minor merge conflicts need resolving. also still to do: merge to lucene_solr_5_3 or move this issue out of the 5.3 CHANGES.txt section ( Shalin Shekhar Mangar just created the 5.4 section - thank you!)
          Hide
          ASF subversion and git services added a comment -

          Commit 1694659 from Christine Poerschke in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1694659 ]

          SOLR-7766: support creation of a coreless collection via createNodeSet=EMPTY (trunk svn r1694181)

          Show
          ASF subversion and git services added a comment - Commit 1694659 from Christine Poerschke in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1694659 ] SOLR-7766 : support creation of a coreless collection via createNodeSet=EMPTY (trunk svn r1694181)
          Hide
          ASF subversion and git services added a comment -

          Commit 1694710 from Christine Poerschke in branch 'dev/branches/lucene_solr_5_3'
          [ https://svn.apache.org/r1694710 ]

          svn merge for revision 1694659 from branch_5x (corresponding to revision 1694181 from trunk)
          SOLR-7766: support creation of a coreless collection via createNodeSet=EMPTY

          Show
          ASF subversion and git services added a comment - Commit 1694710 from Christine Poerschke in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1694710 ] svn merge for revision 1694659 from branch_5x (corresponding to revision 1694181 from trunk) SOLR-7766 : support creation of a coreless collection via createNodeSet=EMPTY
          Hide
          Christine Poerschke added a comment -

          Change committed to trunk, branch_5x and lucene_solr_5_3 with https://cwiki.apache.org/confluence/display/solr/Collections+API?src=search edit to follow shortly.

          Show
          Christine Poerschke added a comment - Change committed to trunk, branch_5x and lucene_solr_5_3 with https://cwiki.apache.org/confluence/display/solr/Collections+API?src=search edit to follow shortly.
          Hide
          Christine Poerschke added a comment -

          https://cwiki.apache.org/confluence/display/solr/Collections+API?src=search#CollectionsAPI-api1 updated, feedback or edits on the change welcome. Would be resolving this ticket here early next week then if all is and remains well.

          Show
          Christine Poerschke added a comment - https://cwiki.apache.org/confluence/display/solr/Collections+API?src=search#CollectionsAPI-api1 updated, feedback or edits on the change welcome. Would be resolving this ticket here early next week then if all is and remains well.
          Hide
          Shalin Shekhar Mangar added a comment -

          Bulk move to 5.4 after 5.3 release.

          Show
          Shalin Shekhar Mangar added a comment - Bulk move to 5.4 after 5.3 release.
          Hide
          Christine Poerschke added a comment -

          Was actually included in 5.3 but i'd just forgotten to close out this ticket, apologies.

          Show
          Christine Poerschke added a comment - Was actually included in 5.3 but i'd just forgotten to close out this ticket, apologies.
          Hide
          ASF GitHub Bot added a comment -

          Github user cpoerschke closed the pull request at:

          https://github.com/apache/lucene-solr/pull/186

          Show
          ASF GitHub Bot added a comment - Github user cpoerschke closed the pull request at: https://github.com/apache/lucene-solr/pull/186

            People

            • Assignee:
              Christine Poerschke
              Reporter:
              Christine Poerschke
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development