Solr
  1. Solr
  2. SOLR-6448

Improve SolrJ support for Collections API

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Right now SolrJ doesn't really support all of the collections API.
      This is a parent issue for bringing SolrJ support for all APIs up to where it should be.

      1. SOLR-6448.patch
        47 kB
        Anshum Gupta
      2. SOLR-6448.patch
        43 kB
        Anshum Gupta
      3. SOLR-6448.patch
        38 kB
        Anshum Gupta
      4. SOLR-6448.patch
        26 kB
        Anshum Gupta
      5. SOLR-6448.patch
        9 kB
        Anshum Gupta

        Issue Links

          Activity

          Hide
          Varun Thacker added a comment -

          A few things that I noticed when looking at this -

          1. The async API call is only supported in the CREATE SolrJ call. It's supported in all of the collection APIs right? We should support it in all of it's equivalent SolrJ calls then
          2. In SOLR-6073 we moved to a builder pattern. We should make the collection api calls immutable

          For example this is what the API calls should look like

              new Create.CreateBuilder("testasynccollectioncreation")
                  .numShards(1)
                  .configName("conf1")
                  .asyncId("1001")
                  .build().process(server);
          

          We can then have all the variables in the Create class be final.

          Show
          Varun Thacker added a comment - A few things that I noticed when looking at this - 1. The async API call is only supported in the CREATE SolrJ call. It's supported in all of the collection APIs right? We should support it in all of it's equivalent SolrJ calls then 2. In SOLR-6073 we moved to a builder pattern. We should make the collection api calls immutable For example this is what the API calls should look like new Create.CreateBuilder( "testasynccollectioncreation" ) .numShards(1) .configName( "conf1" ) .asyncId( "1001" ) .build().process(server); We can then have all the variables in the Create class be final.
          Hide
          Anshum Gupta added a comment -

          The async API call is only supported in the CREATE SolrJ call.

          It is actually supported by CREATE, DELETESHARD, ADDREPLICA (I think every command that supports Async). Also, not all APIs support Async as of now (though I think most of it ideally should).

          We should make the collection api calls immutable

          That would translate to a back compat break. Not sure if we want that.

          Show
          Anshum Gupta added a comment - The async API call is only supported in the CREATE SolrJ call. It is actually supported by CREATE, DELETESHARD, ADDREPLICA (I think every command that supports Async). Also, not all APIs support Async as of now (though I think most of it ideally should). We should make the collection api calls immutable That would translate to a back compat break. Not sure if we want that.
          Hide
          Anshum Gupta added a comment -

          About asyncid support, you're partially right.
          Not all of the calls I mentioned above support AsyncId in SolrJ but a few do. e.g.
          CollectionAdminRequest.CreateShard.getParams() calls getCommonParams which handles AsyncId.

          Show
          Anshum Gupta added a comment - About asyncid support, you're partially right. Not all of the calls I mentioned above support AsyncId in SolrJ but a few do. e.g. CollectionAdminRequest.CreateShard.getParams() calls getCommonParams which handles AsyncId.
          Hide
          Anshum Gupta added a comment -

          Initial patch that adds SolrJ support for:

          • DeleteReplica
          • ClusterProp
          • Migrate
          • AddRole
          • RemoveRole
          • OverseerStatus
          • ClusterStatus
          • List
          • AddReplicaProp
          • DeleteReplicaProp
          • BalanceShardUnique
          • RebalanceLeaders

          Working on the tests. Should have those tomorrow.
          This isn't perhaps the most optimal (best designed) way to do this but it gets us a public API. I want to add basic checks etc. in here to fail fast in case of trivial problems with the request.

          Show
          Anshum Gupta added a comment - Initial patch that adds SolrJ support for: DeleteReplica ClusterProp Migrate AddRole RemoveRole OverseerStatus ClusterStatus List AddReplicaProp DeleteReplicaProp BalanceShardUnique RebalanceLeaders Working on the tests. Should have those tomorrow. This isn't perhaps the most optimal (best designed) way to do this but it gets us a public API. I want to add basic checks etc. in here to fail fast in case of trivial problems with the request.
          Hide
          Anshum Gupta added a comment -

          Updated patch with tests. Need to clean up a few things, might have to restructure a bit but it's almost done for this bit.
          I haven't created any specific CollectionAdminResponse sub-types, specific to the request. Will take that up in the next phase (if I can't get to that by the new year).

          Show
          Anshum Gupta added a comment - Updated patch with tests. Need to clean up a few things, might have to restructure a bit but it's almost done for this bit. I haven't created any specific CollectionAdminResponse sub-types, specific to the request. Will take that up in the next phase (if I can't get to that by the new year).
          Hide
          Anshum Gupta added a comment -

          Still trying to wrap up one test (BALANCESHARDUNIQUE). In the middle of validating that all supported SolrJ methods (for each of the Collection API calls) are actually supported. Once that's done, should be able to commit it and with SolrJ supporting all Collection API calls, it should be a merry christmas!

          Show
          Anshum Gupta added a comment - Still trying to wrap up one test (BALANCESHARDUNIQUE). In the middle of validating that all supported SolrJ methods (for each of the Collection API calls) are actually supported. Once that's done, should be able to commit it and with SolrJ supporting all Collection API calls, it should be a merry christmas!
          Hide
          Shalin Shekhar Mangar added a comment -

          Still trying to wrap up one test (BALANCESHARDUNIQUE).

          Isn't that and REBALANCELEADERS no longer in 5.0? See SOLR-6691

          Show
          Shalin Shekhar Mangar added a comment - Still trying to wrap up one test (BALANCESHARDUNIQUE). Isn't that and REBALANCELEADERS no longer in 5.0? See SOLR-6691
          Hide
          Erick Erickson added a comment -

          BALANCESHARDUNIQUE works fine in 5.0; it correctly updates the cluster state so I left it in 5.0. It may not be useful since the use-case I intended it for is REBALANCELEADERS, but it's still there.

          REBALANCELEADERS OTOH, was pulled from 5.0.

          I put REBALANCELEADERS back in to trunk last night (along with a bad svn EOL style, siiiiggggh) and I'll back-port after 5.0 is cut.

          Anshum Gupta There are a bunch of tests in TestReplicaProperties that might be be useful models. Those tests do the functionality pretty thoroughly, maybe it would be enough for the SolrJ tests to do more minimal testing? Also ReplicaPropertiesBase has some helper methods that you might find useful.

          Show
          Erick Erickson added a comment - BALANCESHARDUNIQUE works fine in 5.0; it correctly updates the cluster state so I left it in 5.0. It may not be useful since the use-case I intended it for is REBALANCELEADERS, but it's still there. REBALANCELEADERS OTOH, was pulled from 5.0. I put REBALANCELEADERS back in to trunk last night (along with a bad svn EOL style, siiiiggggh) and I'll back-port after 5.0 is cut. Anshum Gupta There are a bunch of tests in TestReplicaProperties that might be be useful models. Those tests do the functionality pretty thoroughly, maybe it would be enough for the SolrJ tests to do more minimal testing? Also ReplicaPropertiesBase has some helper methods that you might find useful.
          Hide
          Anshum Gupta added a comment -

          Shalin, as Erick mentioned, REBALANCELEADER was not in there but BALANCESHARDUNIQUE is.
          That list that I put up with my comment is wrong as I never put in RebalanceLeaders in there. The rest is all fine.

          Considering, REBALANCELEADER is only in trunk now, I'll leave that out for now and open another issue for just that and handle it after 5x is out.

          Show
          Anshum Gupta added a comment - Shalin, as Erick mentioned, REBALANCELEADER was not in there but BALANCESHARDUNIQUE is. That list that I put up with my comment is wrong as I never put in RebalanceLeaders in there. The rest is all fine. Considering, REBALANCELEADER is only in trunk now, I'll leave that out for now and open another issue for just that and handle it after 5x is out.
          Hide
          Erick Erickson added a comment -

          Works for me, thanks!

          Show
          Erick Erickson added a comment - Works for me, thanks!
          Hide
          Anshum Gupta added a comment -

          Final patch.

          Show
          Anshum Gupta added a comment - Final patch.
          Hide
          Anshum Gupta added a comment -

          A few more test fixes and a back-compat fix too i.e. deprecating a public call instead of direct removal.

          Show
          Anshum Gupta added a comment - A few more test fixes and a back-compat fix too i.e. deprecating a public call instead of direct removal.
          Hide
          ASF subversion and git services added a comment -

          Commit 1648446 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1648446 ]

          SOLR-6448: SolrJ support for all Collection API calls

          Show
          ASF subversion and git services added a comment - Commit 1648446 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1648446 ] SOLR-6448 : SolrJ support for all Collection API calls
          Hide
          ASF subversion and git services added a comment -

          Commit 1648447 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1648447 ]

          SOLR-6448: SolrJ support for all Collection API calls (merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1648447 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1648447 ] SOLR-6448 : SolrJ support for all Collection API calls (merge from trunk)
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Anshum Gupta
              Reporter:
              Anshum Gupta
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development