Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.5, 5.0
    • Component/s: None
    • Labels:
      None

      Description

      Features to let users control everything about sharding/routing.

      1. SOLR-4221.patch
        2 kB
        Yonik Seeley
      2. SOLR-4221.patch
        16 kB
        Noble Paul
      3. SOLR-4221.patch
        15 kB
        Noble Paul
      4. SOLR-4221.patch
        76 kB
        Noble Paul
      5. SOLR-4221.patch
        75 kB
        Noble Paul
      6. SOLR-4221.patch
        74 kB
        Noble Paul
      7. SOLR-4221.patch
        61 kB
        Noble Paul
      8. SOLR-4221.patch
        0.7 kB
        Noble Paul

        Issue Links

          Activity

          Hide
          Adrien Grand added a comment -

          4.5 release -> bulk close

          Show
          Adrien Grand added a comment - 4.5 release -> bulk close
          Hide
          Markus Jelsma added a comment -

          Adrien, SolrJ can now index to an older cluster successfully! Thanks Yonik, Noble!
          This is great!

          Show
          Markus Jelsma added a comment - Adrien, SolrJ can now index to an older cluster successfully! Thanks Yonik, Noble! This is great!
          Hide
          Adrien Grand added a comment -

          Thanks for doing these tests Markus! I'll wait for your confirmation that the issue is fixed before building new artifacts.

          Show
          Adrien Grand added a comment - Thanks for doing these tests Markus! I'll wait for your confirmation that the issue is fixed before building new artifacts.
          Hide
          Markus Jelsma added a comment -

          Ah, perhaps the current 5.0-SNAPSHOT doesn't yet contain the fix. I'll try again a bit later.

          Show
          Markus Jelsma added a comment - Ah, perhaps the current 5.0-SNAPSHOT doesn't yet contain the fix. I'll try again a bit later.
          Hide
          Noble Paul added a comment -

          Did u test with the newest build? The line numbers aere referring to old build

          Show
          Noble Paul added a comment - Did u test with the newest build? The line numbers aere referring to old build
          Hide
          Markus Jelsma added a comment -

          I tested it again with a SolrJ of just now and an older cluster 18th of september. It keeps throwing the exception:

          java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
                  at org.apache.solr.common.cloud.DocRouter.getRouteField(DocRouter.java:54)
                  at org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:64)
                  at org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33)
                  at org.apache.solr.client.solrj.request.UpdateRequest.getRoutes(UpdateRequest.java:190)
                  at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:313)
                  at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:506)
                  at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
          

          Due to the commit message i assumed it would work now.

          Show
          Markus Jelsma added a comment - I tested it again with a SolrJ of just now and an older cluster 18th of september. It keeps throwing the exception: java.lang.ClassCastException: java.lang. String cannot be cast to java.util.Map at org.apache.solr.common.cloud.DocRouter.getRouteField(DocRouter.java:54) at org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:64) at org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33) at org.apache.solr.client.solrj.request.UpdateRequest.getRoutes(UpdateRequest.java:190) at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:313) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:506) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) Due to the commit message i assumed it would work now.
          Hide
          ASF subversion and git services added a comment -

          Commit 1526395 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1526395 ]

          SOLR-4221 make new solrj client/router able to read old clusterstate

          Show
          ASF subversion and git services added a comment - Commit 1526395 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1526395 ] SOLR-4221 make new solrj client/router able to read old clusterstate
          Hide
          ASF subversion and git services added a comment -

          Commit 1526255 from Yonik Seeley in branch 'dev/branches/lucene_solr_4_5'
          [ https://svn.apache.org/r1526255 ]

          SOLR-4221: make new solrj client/router able to read old clusterstate

          Show
          ASF subversion and git services added a comment - Commit 1526255 from Yonik Seeley in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1526255 ] SOLR-4221 : make new solrj client/router able to read old clusterstate
          Hide
          ASF subversion and git services added a comment -

          Commit 1526244 from Yonik Seeley in branch 'dev/trunk'
          [ https://svn.apache.org/r1526244 ]

          SOLR-4221: make new solrj client/router able to read old clusterstate

          Show
          ASF subversion and git services added a comment - Commit 1526244 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1526244 ] SOLR-4221 : make new solrj client/router able to read old clusterstate
          Hide
          Noble Paul added a comment - - edited

          Thanks Markus Jelsma good catch

          I'll connect it to SOLR-5258

          Show
          Noble Paul added a comment - - edited Thanks Markus Jelsma good catch I'll connect it to SOLR-5258
          Hide
          Markus Jelsma added a comment -

          Ah, i come from SOLR-5261 and noticed the discussion here so i'll comment here as well. The stack trace i posted earlier is gone now and is replaced by the trace below. This only happens if a new SolrJ attempts to talk to a slightly older cluster.

          java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
                  at org.apache.solr.common.cloud.DocRouter.getRouteField(DocRouter.java:54)
                  at org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:64)
                  at org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33)
                  at org.apache.solr.client.solrj.request.UpdateRequest.getRoutes(UpdateRequest.java:190)
                  at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:313)
                  at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:506)
                  at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
          
          Show
          Markus Jelsma added a comment - Ah, i come from SOLR-5261 and noticed the discussion here so i'll comment here as well. The stack trace i posted earlier is gone now and is replaced by the trace below. This only happens if a new SolrJ attempts to talk to a slightly older cluster. java.lang.ClassCastException: java.lang. String cannot be cast to java.util.Map at org.apache.solr.common.cloud.DocRouter.getRouteField(DocRouter.java:54) at org.apache.solr.common.cloud.CompositeIdRouter.sliceHash(CompositeIdRouter.java:64) at org.apache.solr.common.cloud.HashBasedRouter.getTargetSlice(HashBasedRouter.java:33) at org.apache.solr.client.solrj.request.UpdateRequest.getRoutes(UpdateRequest.java:190) at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:313) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:506) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
          Hide
          Noble Paul added a comment -

          Yeah , that works

          But if my usecase is,

          Bring down each node , replace the jar and restart the node this will fail. right?

          If this does not work , we will have to document it clearly

          Show
          Noble Paul added a comment - Yeah , that works But if my usecase is, Bring down each node , replace the jar and restart the node this will fail. right? If this does not work , we will have to document it clearly
          Hide
          Yonik Seeley added a comment -

          I don't think we can fix this in a a backcompat manner

          There are different degrees of back compat, and I've already committed something that at least works a lot better than what was there. I tested by creating a 4.4 cluster, bringing it down, and then replacing the jar file with 4.5.

          Show
          Yonik Seeley added a comment - I don't think we can fix this in a a backcompat manner There are different degrees of back compat, and I've already committed something that at least works a lot better than what was there. I tested by creating a 4.4 cluster, bringing it down, and then replacing the jar file with 4.5.
          Hide
          Noble Paul added a comment -

          Yonik Seeley I don't think we can fix this in a a backcompat manner if we reuse the name router and change the Object type.

          There will be atleast one node in the cluster which will have the old binaries and will expect the router instanceof String. the new binaries will keep writing router as Map.

          The backward compatible solution would be to choose another name say "route" and if the 'router' value is missing, the old binaries would not complain

          Show
          Noble Paul added a comment - Yonik Seeley I don't think we can fix this in a a backcompat manner if we reuse the name router and change the Object type. There will be atleast one node in the cluster which will have the old binaries and will expect the router instanceof String. the new binaries will keep writing router as Map. The backward compatible solution would be to choose another name say "route" and if the 'router' value is missing, the old binaries would not complain
          Hide
          ASF subversion and git services added a comment -

          Commit 1526008 from Yonik Seeley in branch 'dev/branches/lucene_solr_4_5'
          [ https://svn.apache.org/r1526008 ]

          SOLR-4221: back compat for router spec

          Show
          ASF subversion and git services added a comment - Commit 1526008 from Yonik Seeley in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1526008 ] SOLR-4221 : back compat for router spec
          Hide
          ASF subversion and git services added a comment -

          Commit 1526007 from Yonik Seeley in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1526007 ]

          SOLR-4221: back compat for router spec

          Show
          ASF subversion and git services added a comment - Commit 1526007 from Yonik Seeley in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1526007 ] SOLR-4221 : back compat for router spec
          Hide
          ASF subversion and git services added a comment -

          Commit 1526003 from Yonik Seeley in branch 'dev/trunk'
          [ https://svn.apache.org/r1526003 ]

          SOLR-4221: back compat for router spec

          Show
          ASF subversion and git services added a comment - Commit 1526003 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1526003 ] SOLR-4221 : back compat for router spec
          Hide
          Yonik Seeley added a comment -

          Here's a patch that seems to work with some manual back compat testing. I'll commit after the unit tests run.

          Show
          Yonik Seeley added a comment - Here's a patch that seems to work with some manual back compat testing. I'll commit after the unit tests run.
          Hide
          Yonik Seeley added a comment -

          I think the last patch here broke back compat with clusterstate (Markus hit this, reported in SOLR-5261)
          I'll try to come up with a patch. It shouldn't be too hard to fix (longest part will be testing).

          Show
          Yonik Seeley added a comment - I think the last patch here broke back compat with clusterstate (Markus hit this, reported in SOLR-5261 ) I'll try to come up with a patch. It shouldn't be too hard to fix (longest part will be testing).
          Hide
          ASF subversion and git services added a comment -

          Commit 1524742 from Adrien Grand in branch 'dev/branches/lucene_solr_4_5'
          [ https://svn.apache.org/r1524742 ]

          SOLR-4221 changed format router is stored

          Show
          ASF subversion and git services added a comment - Commit 1524742 from Adrien Grand in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1524742 ] SOLR-4221 changed format router is stored
          Hide
          ASF subversion and git services added a comment -

          Commit 1524446 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1524446 ]

          SOLR-4221 changed format router is stored

          Show
          ASF subversion and git services added a comment - Commit 1524446 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1524446 ] SOLR-4221 changed format router is stored
          Hide
          ASF subversion and git services added a comment -

          Commit 1524442 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1524442 ]

          SOLR-4221 changed format router is stored

          Show
          ASF subversion and git services added a comment - Commit 1524442 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1524442 ] SOLR-4221 changed format router is stored
          Hide
          Noble Paul added a comment -

          fixed some failing tests

          Show
          Noble Paul added a comment - fixed some failing tests
          Hide
          Noble Paul added a comment -

          router and associated variables moved to an object

          Show
          Noble Paul added a comment - router and associated variables moved to an object
          Hide
          Noble Paul added a comment -

          Makes sense

          when the collection is created let us accept the parameters with the 'router' prefix . eg: router.name=implicit&router.field=somefield. So any value that has a router prefix will go into the router attribute. we can use it in the future for more attributes and even custom attributes for custom routers. eg: router.class=my.ClassName&router.attr1=x&router.attr2=y

          when it is persisted, it can be

          router :

          {"name":"implicit","field":"somefield"}
          Show
          Noble Paul added a comment - Makes sense when the collection is created let us accept the parameters with the 'router' prefix . eg: router.name=implicit&router.field=somefield. So any value that has a router prefix will go into the router attribute. we can use it in the future for more attributes and even custom attributes for custom routers. eg: router.class=my.ClassName&router.attr1=x&router.attr2=y when it is persisted, it can be router : {"name":"implicit","field":"somefield"}
          Hide
          Yonik Seeley added a comment - - edited

          Hmmm, it feels like the "routeField" parameter should be scoped under "router" somehow... I think there will be additional parameters to configure a router in the future (such as number of bits to allocate to parts of the compositeId router, etc), as well as custom routers, where their configuration could include additional parameters that would best be scoped.

          router.routeField?

          As far as persistence, it could be flat, but perhaps nicer to scope that as well...

          "router" :

          {"name" : "implicit", "routField" : "companyName"}
          Show
          Yonik Seeley added a comment - - edited Hmmm, it feels like the "routeField" parameter should be scoped under "router" somehow... I think there will be additional parameters to configure a router in the future (such as number of bits to allocate to parts of the compositeId router, etc), as well as custom routers, where their configuration could include additional parameters that would best be scoped. router.routeField? As far as persistence, it could be flat, but perhaps nicer to scope that as well... "router" : {"name" : "implicit", "routField" : "companyName"}
          Hide
          Hoss Man added a comment -

          Is this feature complete? (all of the subtasks are marked resolved and several commits associated with this isue are in branch 4x)

          Can someone who understands all of the various changes made in these issues please update the ref guide (or post a comment suggestion what additions should be made)...

          https://cwiki.apache.org/confluence/display/solr/Collections+API

          Show
          Hoss Man added a comment - Is this feature complete? (all of the subtasks are marked resolved and several commits associated with this isue are in branch 4x) Can someone who understands all of the various changes made in these issues please update the ref guide (or post a comment suggestion what additions should be made)... https://cwiki.apache.org/confluence/display/solr/Collections+API
          Hide
          ASF subversion and git services added a comment -

          Commit 1509017 from shalin@apache.org in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1509017 ]

          SOLR-4221: Fix compile error on Java6 due to use of diamond operator

          Show
          ASF subversion and git services added a comment - Commit 1509017 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1509017 ] SOLR-4221 : Fix compile error on Java6 due to use of diamond operator
          Hide
          Noble Paul added a comment -

          A custom collection can be created by explicitly specifying the router as 'implicit' and by giving the names of the shards explicitly as follows

          http://host:port/solr/admin/collections?action=CREATE&name=monthly&router=implicit&collection.configName=x&shards=jan13,feb13,mar13

          where 3 shards named jan13,feb13,mar13 will be created

          Document routing

          • add a special request param called _route_=june13’ . It means all docs added in the batch would belong to the same shard or,
          • add a special field _route_ in the document to signal the shard where it belongs to
          • if the collection is created with ‘routeField’ value , the docRouter would look at that field to get the shard name SOLR-5017
          • If it is unable to identify the shard using any of the above methods , the write operation fails. If the shard is not present , then the write fails too

          Query routing

          It remain same as other routers. _route_ parameter can specify the actual shard name

          Creating new shards

          There is a a new command CREATESHARD . refer to SOLR-5006.

          Show
          Noble Paul added a comment - A custom collection can be created by explicitly specifying the router as 'implicit' and by giving the names of the shards explicitly as follows http://host:port/solr/admin/collections?action=CREATE&name=monthly&router=implicit&collection.configName=x&shards=jan13,feb13,mar13 where 3 shards named jan13,feb13,mar13 will be created Document routing add a special request param called _route_=june13’ . It means all docs added in the batch would belong to the same shard or, add a special field _route_ in the document to signal the shard where it belongs to if the collection is created with ‘routeField’ value , the docRouter would look at that field to get the shard name SOLR-5017 If it is unable to identify the shard using any of the above methods , the write operation fails. If the shard is not present , then the write fails too Query routing It remain same as other routers. _route_ parameter can specify the actual shard name Creating new shards There is a a new command CREATESHARD . refer to SOLR-5006 .
          Hide
          ASF subversion and git services added a comment -

          Commit 1508981 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1508981 ]

          SOLR-4221 SOLR-4808 SOLR-5006 SOLR-5017 SOLR-4222

          Show
          ASF subversion and git services added a comment - Commit 1508981 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1508981 ] SOLR-4221 SOLR-4808 SOLR-5006 SOLR-5017 SOLR-4222
          Hide
          ASF subversion and git services added a comment -
          Show
          ASF subversion and git services added a comment - Commit 1508968 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1508968 ] SOLR-4221 SOLR-4808 SOLR-5006 SOLR-5017 SOLR-4222
          Hide
          Anshum Gupta added a comment -

          Spoke to Noble offline and perhaps it's fine to deliberately not check it in OCP and just in the CollectionsHandler.
          This lets Shards be created even when the router is non-implicit for cases like ShardSplitting.

          Show
          Anshum Gupta added a comment - Spoke to Noble offline and perhaps it's fine to deliberately not check it in OCP and just in the CollectionsHandler. This lets Shards be created even when the router is non-implicit for cases like ShardSplitting.
          Hide
          Anshum Gupta added a comment -

          You may want to check on the current router being used, before being able to add a shard.

          Show
          Anshum Gupta added a comment - You may want to check on the current router being used, before being able to add a shard.
          Hide
          Noble Paul added a comment -

          tests added for CREATESHARD

          Show
          Noble Paul added a comment - tests added for CREATESHARD
          Hide
          Noble Paul added a comment -

          I plan to commit this soon

          Show
          Noble Paul added a comment - I plan to commit this soon
          Hide
          Noble Paul added a comment -

          OverseerCollectionProcessor test errors fixed

          Show
          Noble Paul added a comment - OverseerCollectionProcessor test errors fixed
          Hide
          Noble Paul added a comment -

          working patch with testcases .

          Show
          Noble Paul added a comment - working patch with testcases .
          Hide
          Noble Paul added a comment -

          The working code is posted here https://github.com/shalinmangar/lucene-solr . We will start moving it to SVN after 4.4 branching

          Show
          Noble Paul added a comment - The working code is posted here https://github.com/shalinmangar/lucene-solr . We will start moving it to SVN after 4.4 branching
          Hide
          Shai Erera added a comment -

          Exactly. New nodes mean just more resources, I don't think they should affect the number of resources the search cluster needs, only define how many resources are available. Conceptually, we could have a cost function which takes into account the node's RAM/CPU/Disk + indexes size (GBs, #docs), maybe even based on performance history, and assign indexes to nodes following that cost function.

          The way I picture it (and I don't know how to map that to Solr jargon):

          • A new node comes up, broadcasts "I'm alive"
          • Maybe communicates with other "overbooked" nodes to transfer indexes over
          • A ClusterBalancer (Overseer?) periodically checks if the cluster can be better balanced (e.g. splitting shards, moving indexes)
          • Maybe even we drop numShards entirely and make this a completely dynamic and automatic setting (or at least optional), but this is for future work
          Show
          Shai Erera added a comment - Exactly. New nodes mean just more resources, I don't think they should affect the number of resources the search cluster needs, only define how many resources are available. Conceptually, we could have a cost function which takes into account the node's RAM/CPU/Disk + indexes size (GBs, #docs), maybe even based on performance history, and assign indexes to nodes following that cost function. The way I picture it (and I don't know how to map that to Solr jargon): A new node comes up, broadcasts "I'm alive" Maybe communicates with other "overbooked" nodes to transfer indexes over A ClusterBalancer (Overseer?) periodically checks if the cluster can be better balanced (e.g. splitting shards, moving indexes) Maybe even we drop numShards entirely and make this a completely dynamic and automatic setting (or at least optional), but this is for future work
          Hide
          Noble Paul added a comment - - edited

          Another usecase is to have multiple collections, each with different number of shards and different #replicas.

          This is why I feel we should not automatically create cores in nodes that come up. The best thing to do is

          Any new node will just not participate in any collection (unless a shard has fewer nodes than replicationFactor)

          There should be an explicit ASSIGN_NODE command to add/remove nodes to/from a shard.
          It should be possible to do an ASSIGN_NODE without specifying a nodename in which case the overseer would look for free nodes in the cluster and add to the specified shard

          when a new shard is created by CREATESHARD command or by a SPLITSHARD command these nodes could be automatically be taken up.

          Show
          Noble Paul added a comment - - edited Another usecase is to have multiple collections, each with different number of shards and different #replicas. This is why I feel we should not automatically create cores in nodes that come up. The best thing to do is Any new node will just not participate in any collection (unless a shard has fewer nodes than replicationFactor) There should be an explicit ASSIGN_NODE command to add/remove nodes to/from a shard. It should be possible to do an ASSIGN_NODE without specifying a nodename in which case the overseer would look for free nodes in the cluster and add to the specified shard when a new shard is created by CREATESHARD command or by a SPLITSHARD command these nodes could be automatically be taken up.
          Hide
          Yonik Seeley added a comment -

          Another usecase is to have multiple collections, each with different number of shards and different #replicas.

          Right - we need to figure out how to accomodate all these use cases.

          Show
          Yonik Seeley added a comment - Another usecase is to have multiple collections, each with different number of shards and different #replicas. Right - we need to figure out how to accomodate all these use cases.
          Hide
          Shai Erera added a comment -

          Another usecase is to have multiple collections, each with different number of shards and different #replicas. I would not want to see 10 replicas for each shard just because I loaded 10 servers. Rather, I think it would be useful if I could say "collection1, #shards=3, #replicas=2" (total 6 indexes) and "collection2, #shards=4, #replicas=5" (total 20 indexes) and if I bring up a 10 nodes cluster, the indexes are somehow distributed across them.

          Show
          Shai Erera added a comment - Another usecase is to have multiple collections, each with different number of shards and different #replicas. I would not want to see 10 replicas for each shard just because I loaded 10 servers. Rather, I think it would be useful if I could say "collection1, #shards=3, #replicas=2" (total 6 indexes) and "collection2, #shards=4, #replicas=5" (total 20 indexes) and if I bring up a 10 nodes cluster, the indexes are somehow distributed across them.
          Hide
          Yonik Seeley added a comment -

          does it also mean if ever the no:of replicas fall below the 'minReplicas' , write ops would fail?

          I was considering it only as a create-time parameter (not persisted), and had originally thought about a parameter on updates that could specify the required replication level for safety. But we really need to think about all the use-cases we can up front - it could make sense to have additional persisted properties around replicationFactor.

          One use case is expanding your search capacity dynamically by simply starting up new nodes that automatically become replicas of existing shards. This is the current behavior today. We need to think about if/how to handle this use case when we start persisting replicationFactor and if the overseer will destroy nodes to bring things back to the replication factor. One way would be to set the replicationFactor really high (to prevent this distruction), but this could lead to other negatives such as a future GUI displaying the status of the collection as degraded since the replicationFactor isn't met. Requiring the user to bump the replicationFactor up before adding new nodes isn't the friendliest thing given that this will be a very common use case.

          Show
          Yonik Seeley added a comment - does it also mean if ever the no:of replicas fall below the 'minReplicas' , write ops would fail? I was considering it only as a create-time parameter (not persisted), and had originally thought about a parameter on updates that could specify the required replication level for safety. But we really need to think about all the use-cases we can up front - it could make sense to have additional persisted properties around replicationFactor. One use case is expanding your search capacity dynamically by simply starting up new nodes that automatically become replicas of existing shards. This is the current behavior today. We need to think about if/how to handle this use case when we start persisting replicationFactor and if the overseer will destroy nodes to bring things back to the replication factor. One way would be to set the replicationFactor really high (to prevent this distruction), but this could lead to other negatives such as a future GUI displaying the status of the collection as degraded since the replicationFactor isn't met. Requiring the user to bump the replicationFactor up before adding new nodes isn't the friendliest thing given that this will be a very common use case.
          Hide
          Noble Paul added a comment -

          If I specify replicationFactor=2 and minReplicas=1

          and Ithe cluster could create only 1 core , What is the state of the shard? ACTIVE|INACTIVE or something new ?

          Show
          Noble Paul added a comment - If I specify replicationFactor=2 and minReplicas=1 and Ithe cluster could create only 1 core , What is the state of the shard? ACTIVE|INACTIVE or something new ?
          Hide
          Noble Paul added a comment -

          I agree with you Yonik. forceCreate is a superset and can mean a lot of things.

          minReplicas make sense.

          But we need a clear definition of the following . Otherwise we are getting into a slippery slope

          • replicationFactor : I'm still not very clear about this because it has a different meanoing in other nosql systems.
          • minReplicas

          does it also mean if ever the no:of replicas fall below the 'minReplicas' , write ops would fail? Or does it only mean that it is only for creation of cluster/shard ?

          we should also go ahead and define what those will mean when the 'autoManageCluster' (or whatever it is going to be called) is implemented .

          Show
          Noble Paul added a comment - I agree with you Yonik. forceCreate is a superset and can mean a lot of things. minReplicas make sense. But we need a clear definition of the following . Otherwise we are getting into a slippery slope replicationFactor : I'm still not very clear about this because it has a different meanoing in other nosql systems. minReplicas does it also mean if ever the no:of replicas fall below the 'minReplicas' , write ops would fail? Or does it only mean that it is only for creation of cluster/shard ? we should also go ahead and define what those will mean when the 'autoManageCluster' (or whatever it is going to be called) is implemented .
          Hide
          Yonik Seeley added a comment -

          Do we really need to have so many configuration params and complicate this?

          Eh, you replaced one parameter (a minimum replication factor to achieve) with another (a boolean meaning 0 or replicationFactor).
          This seems akin to facet.includeZeros vs facet.minCount... the latter incorporates the former and is more powerful (hence fewer parameters in the long run), and the former would have never existed if the latter had been thought of first.

          Now, I don't like the name "createReplictionFactor" (maybe "minReplicas" or something like that would be better), but the functionality is a superset of "forceCreate" and is more descriptive (after all, there are many ways to force things). It also seems useful to say something like "create this collection and make sure it's usable (at least one replica for every shard) but don't worry about trying to satisfy the replciationFactor just yet). That would be minReplicas=1

          Show
          Yonik Seeley added a comment - Do we really need to have so many configuration params and complicate this? Eh, you replaced one parameter (a minimum replication factor to achieve) with another (a boolean meaning 0 or replicationFactor). This seems akin to facet.includeZeros vs facet.minCount... the latter incorporates the former and is more powerful (hence fewer parameters in the long run), and the former would have never existed if the latter had been thought of first. Now, I don't like the name "createReplictionFactor" (maybe "minReplicas" or something like that would be better), but the functionality is a superset of "forceCreate" and is more descriptive (after all, there are many ways to force things). It also seems useful to say something like "create this collection and make sure it's usable (at least one replica for every shard) but don't worry about trying to satisfy the replciationFactor just yet). That would be minReplicas=1
          Hide
          Jan Høydahl added a comment -

          If you wish to create a new collection/shard w/o enough nodes we can just use something like forceCreate=true so that the collection and shards are created empty + inactive . if forceCreate=false let us just fail if nodes are insufficient

          +1

          When adding a new empty node, it should only get auto-assigned replicas if any of the existing collections have not filled up its numShards and replicationFactor yet. Else it should be left empty and then be utilized by future create collection requests or rebalance/split requests.

          Show
          Jan Høydahl added a comment - If you wish to create a new collection/shard w/o enough nodes we can just use something like forceCreate=true so that the collection and shards are created empty + inactive . if forceCreate=false let us just fail if nodes are insufficient +1 When adding a new empty node, it should only get auto-assigned replicas if any of the existing collections have not filled up its numShards and replicationFactor yet. Else it should be left empty and then be utilized by future create collection requests or rebalance/split requests.
          Hide
          Noble Paul added a comment -

          As far as consistency with how collection creation works, I think we're going to need an additional parameter now that we can store the the replicationFactor in ZK and it can act as a target.

          Do we really need to have so many configuration params and complicate this? I believe we should define a standard recipe for doing something and people will just follow . Adding more options like createReplictionFactor is not elegant .

          If you wish to create a new collection/shard w/o enough nodes we can just use something like forceCreate=true so that the collection and shards are created empty + inactive . if forceCreate=false let us just fail if nodes are insufficient

          Show
          Noble Paul added a comment - As far as consistency with how collection creation works, I think we're going to need an additional parameter now that we can store the the replicationFactor in ZK and it can act as a target. Do we really need to have so many configuration params and complicate this? I believe we should define a standard recipe for doing something and people will just follow . Adding more options like createReplictionFactor is not elegant . If you wish to create a new collection/shard w/o enough nodes we can just use something like forceCreate=true so that the collection and shards are created empty + inactive . if forceCreate=false let us just fail if nodes are insufficient
          Hide
          Yonik Seeley added a comment -

          If I create the shard first and I don't have enough nodes to sign it to, it should probably just fail

          Seems like it should be fine to create a shard w/ no current replicas (but it should also be fine to specify a certain minimum replication factor for the create to succeed).

          As far as consistency with how collection creation works, I think we're going to need an additional parameter now that we can store the the replicationFactor in ZK and it can act as a target. We need something like createReplicationFactor or something that specifies the minimum replication factor to achieve for the collection creation to succeed.

          Auto rebalancing should be an opt in thing , right?

          It should certainly be optional/configurable. What the default should be is a decision we can tackle once it's implemented.

          Show
          Yonik Seeley added a comment - If I create the shard first and I don't have enough nodes to sign it to, it should probably just fail Seems like it should be fine to create a shard w/ no current replicas (but it should also be fine to specify a certain minimum replication factor for the create to succeed). As far as consistency with how collection creation works, I think we're going to need an additional parameter now that we can store the the replicationFactor in ZK and it can act as a target. We need something like createReplicationFactor or something that specifies the minimum replication factor to achieve for the collection creation to succeed. Auto rebalancing should be an opt in thing , right? It should certainly be optional/configurable. What the default should be is a decision we can tackle once it's implemented.
          Hide
          Noble Paul added a comment -

          If I create the shard first and I don't have enough nodes to sign it to, it should probably just fai just the way create collection does. That would be the consistent behavior

          Auto rebalancing should be an opt in thing , right?

          Show
          Noble Paul added a comment - If I create the shard first and I don't have enough nodes to sign it to, it should probably just fai just the way create collection does. That would be the consistent behavior Auto rebalancing should be an opt in thing , right?
          Hide
          Yonik Seeley added a comment -

          What is the work flow like? Do I add nodes first or do I create the shards/collection first?

          Both?

          Automatically assigning nodes to shards even when we have sufficient replicas is something I find strange

          Yeah, that's sort of a different case... I was sort of assuming that we either didn't have sufficient replicas, or that the overseer may chose to rebalance some replicas off more heavily loaded nodes.

          Show
          Yonik Seeley added a comment - What is the work flow like? Do I add nodes first or do I create the shards/collection first? Both? Automatically assigning nodes to shards even when we have sufficient replicas is something I find strange Yeah, that's sort of a different case... I was sort of assuming that we either didn't have sufficient replicas, or that the overseer may chose to rebalance some replicas off more heavily loaded nodes.
          Hide
          Noble Paul added a comment -

          I miss something here. What is the work flow like? Do I add nodes first or do I create the shards/collection first? I can pass these api params only when the shard/collection is happening. Automatically assigning nodes to shards even when we have sufficient replicas is something I find strange

          Show
          Noble Paul added a comment - I miss something here. What is the work flow like? Do I add nodes first or do I create the shards/collection first? I can pass these api params only when the shard/collection is happening. Automatically assigning nodes to shards even when we have sufficient replicas is something I find strange
          Hide
          Yonik Seeley added a comment -

          The proposed behavior is this. If I startup a node w/o specifying the collection , it would just be a part of the live nodes and no shard and collection should be assigned to it.

          That's not so friendly though... seems like most of the time we would want to just be able to start a new node and have it utilized automatically.

          I think your use case (reserving some nodes for a new collection/shard/whatever you know you are going to create) may be best served by temporarily setting autoCreateReplicas=false (or whatever) via API. If autoCreateReplicas==false, then there should be a way to bring up a node and not have cores created on it.

          Show
          Yonik Seeley added a comment - The proposed behavior is this. If I startup a node w/o specifying the collection , it would just be a part of the live nodes and no shard and collection should be assigned to it. That's not so friendly though... seems like most of the time we would want to just be able to start a new node and have it utilized automatically. I think your use case (reserving some nodes for a new collection/shard/whatever you know you are going to create) may be best served by temporarily setting autoCreateReplicas=false (or whatever) via API. If autoCreateReplicas==false, then there should be a way to bring up a node and not have cores created on it.
          Hide
          Mark Miller added a comment -

          If I startup a node w/o specifying the collection

          To get that behavior we should be starting with no cores defined out of the box - you start up a node and it joins nothing because it has no cores. Then collection create commands and what not can start using that empty node. I don't think we care about starting a core that doesn't belong to a collection - we just want to get to the point where you start a solrcloud node and there is no predefined cores.

          Show
          Mark Miller added a comment - If I startup a node w/o specifying the collection To get that behavior we should be starting with no cores defined out of the box - you start up a node and it joins nothing because it has no cores. Then collection create commands and what not can start using that empty node. I don't think we care about starting a core that doesn't belong to a collection - we just want to get to the point where you start a solrcloud node and there is no predefined cores.
          Hide
          Noble Paul added a comment -

          Ideally a user would want new shards to go to new nodes so that the existing nodes are not overloaded. Now it is not possible to achieve it. If I start a node before t5he command is issued it would just join some random shard and it is taken up. If I issue the CREATESHARD command before adding nodes it will/should fail saying insufficient nodes .

          Actually, it is relevant for SPLITSHARDas well. Now the new shards created will go and live in an already loaded node.

          The proposed behavior is this. If I startup a node w/o specifying the collection , it would just be a part of the live nodes and no shard and collection should be assigned to it. So when a new shard has to be created , the command can lookup and see if there are unutilized live nodes and it can be assigned there

          Show
          Noble Paul added a comment - Ideally a user would want new shards to go to new nodes so that the existing nodes are not overloaded. Now it is not possible to achieve it. If I start a node before t5he command is issued it would just join some random shard and it is taken up. If I issue the CREATESHARD command before adding nodes it will/should fail saying insufficient nodes . Actually, it is relevant for SPLITSHARDas well. Now the new shards created will go and live in an already loaded node. The proposed behavior is this. If I startup a node w/o specifying the collection , it would just be a part of the live nodes and no shard and collection should be assigned to it. So when a new shard has to be created , the command can lookup and see if there are unutilized live nodes and it can be assigned there
          Hide
          Mark Miller added a comment -

          I don't think it's a big deal myself - if it can be improved without adding complication, thats fine - but you can still simply view a CoreDescriptor as information to be used for core creation. I don't know that it ever promised to have 'live' information or something - the SolrCore reads the info on the CoreDescriptor to create the SolrCore.

          It is not 100% satisfying, but unless the solution is pretty simple, I don't think it's worth worrying about near term.

          Show
          Mark Miller added a comment - I don't think it's a big deal myself - if it can be improved without adding complication, thats fine - but you can still simply view a CoreDescriptor as information to be used for core creation. I don't know that it ever promised to have 'live' information or something - the SolrCore reads the info on the CoreDescriptor to create the SolrCore. It is not 100% satisfying, but unless the solution is pretty simple, I don't think it's worth worrying about near term.
          Hide
          Shalin Shekhar Mangar added a comment -

          why do we keep the numShards in the CoreDescriptor? It says " // setting only matters on core creation" .Is it the right place to keep it

          No it is not the right place. Besides numShards, there's shardRange and shardState too which are used only for core creation. We need to figure out where we can keep such properties that are useful only during core creation.

          Show
          Shalin Shekhar Mangar added a comment - why do we keep the numShards in the CoreDescriptor? It says " // setting only matters on core creation" .Is it the right place to keep it No it is not the right place. Besides numShards, there's shardRange and shardState too which are used only for core creation. We need to figure out where we can keep such properties that are useful only during core creation.
          Hide
          Noble Paul added a comment - - edited

          why do we keep the numShards in the CoreDescriptor? It says " // setting only matters on core creation" .Is it the right place to keep it

          when a "state" operation is sent to overseer, other information such as router etc is missing and it is difficult to create collection with any router other than the default

          Show
          Noble Paul added a comment - - edited why do we keep the numShards in the CoreDescriptor? It says " // setting only matters on core creation" .Is it the right place to keep it when a "state" operation is sent to overseer, other information such as router etc is missing and it is difficult to create collection with any router other than the default
          Hide
          Mark Miller added a comment -

          It's only a dupe by text name afaict - really it should be a part tracked as part of this issue IMO.

          Show
          Mark Miller added a comment - It's only a dupe by text name afaict - really it should be a part tracked as part of this issue IMO.
          Hide
          Otis Gospodnetic added a comment -

          Noble Paul should SOLR-4059 be closed as dupe?

          Show
          Otis Gospodnetic added a comment - Noble Paul should SOLR-4059 be closed as dupe?
          Hide
          Mark Miller added a comment -

          Nice tracking that down! It was weird as hell. I could not figure out what I was missing. I think we should fix it as it's own bug - it can go into 4.4 when this may not.

          Show
          Mark Miller added a comment - Nice tracking that down! It was weird as hell. I could not figure out what I was missing. I think we should fix it as it's own bug - it can go into 4.4 when this may not.
          Hide
          Noble Paul added a comment - - edited

          Very interesting. This can go into a java tricks n tips book

            static Integer msgToStr(String val,Integer def ){
                  return  val == null ? def : Integer.parseInt(val);
              }
          

          compiles to

            static Integer msgToStr(String val,Integer def ){
                  return Integer.valueOf(val == null ? def : Integer.parseInt(val));
              }
          

          So NPE , if both 'val' and 'def' are null

          Should I open another bug or just commit it as a SOLR-4221

          Show
          Noble Paul added a comment - - edited Very interesting. This can go into a java tricks n tips book static Integer msgToStr( String val, Integer def ){ return val == null ? def : Integer .parseInt(val); } compiles to static Integer msgToStr( String val, Integer def ){ return Integer .valueOf(val == null ? def : Integer .parseInt(val)); } So NPE , if both 'val' and 'def' are null Should I open another bug or just commit it as a SOLR-4221
          Hide
          Noble Paul added a comment -

          Yes, It gives an NPE if numShards is missing. It should throw a proper message why it is failing. That'll be fixed as a part of this

          Show
          Noble Paul added a comment - Yes, It gives an NPE if numShards is missing. It should throw a proper message why it is failing. That'll be fixed as a part of this
          Hide
          Mark Miller added a comment -

          I get a NPE if I don't provide numShards)

          That's a weird little bug I ran into a couple days ago - you should be getting a message about numShards being required (it is currently).

          Show
          Mark Miller added a comment - I get a NPE if I don't provide numShards) That's a weird little bug I ran into a couple days ago - you should be getting a message about numShards being required (it is currently).
          Hide
          Yonik Seeley added a comment -

          In case of "implicit" router , we do not wish to assign shard names

          Right - I was more talking about the default when we were already using the compositeId router.
          It seems like that for the collections API, defaulting to router=compositeId and numShards=1 is reasonable.

          Is it possible to create collection w/ no shard?, or we do we need to add support for that first?

          I don't think the collections API currently has support for routers other than the default compositeId (just tried it... I get a NPE if I don't provide numShards). There may be issues at a lower level too - not sure (initially, collections were created by bringing up a core - the collections API came later).

          Show
          Yonik Seeley added a comment - In case of "implicit" router , we do not wish to assign shard names Right - I was more talking about the default when we were already using the compositeId router. It seems like that for the collections API, defaulting to router=compositeId and numShards=1 is reasonable. Is it possible to create collection w/ no shard?, or we do we need to add support for that first? I don't think the collections API currently has support for routers other than the default compositeId (just tried it... I get a NPE if I don't provide numShards). There may be issues at a lower level too - not sure (initially, collections were created by bringing up a core - the collections API came later).
          Hide
          Noble Paul added a comment -

          We want things to be as easy as possible to get going... it may make sense to have a default of 1 for numShards.

          In case of "implicit" router , we do not wish to assign shard names . So the parameter is pretty useless. We can only create as many shards as user provides. If no 'shard' params are present we will have to create a collection with no shards

          The update is assumed to belong on the receiving shard (that's the implicit part).

          yeah. makes sense. I can't imagine a name which makes sense foe both cases

          One should be able to create a collection w/ no shards also (although normally the next steps would be to create shards).

          Is it possible to create collection w/ no shard?, or we do we need to add support for that first?

          Show
          Noble Paul added a comment - We want things to be as easy as possible to get going... it may make sense to have a default of 1 for numShards. In case of "implicit" router , we do not wish to assign shard names . So the parameter is pretty useless. We can only create as many shards as user provides. If no 'shard' params are present we will have to create a collection with no shards The update is assumed to belong on the receiving shard (that's the implicit part). yeah. makes sense. I can't imagine a name which makes sense foe both cases One should be able to create a collection w/ no shards also (although normally the next steps would be to create shards). Is it possible to create collection w/ no shard?, or we do we need to add support for that first?
          Hide
          Yonik Seeley added a comment -

          'numShards is a required param for default 'compositeId' router.

          We want things to be as easy as possible to get going... it may make sense to have a default of 1 for numShards.

          The name of the Router is a bit confusing. I fee it should be called "explicit" instead of "implicit". But let us leave it

          Yeah... I guess what I was thinking at the time was when someone sent and update to a shard w/ no further routing info. The update is assumed to belong on the receiving shard (that's the implicit part). Of course it should work with explict shard info too.

          new required parameter 'shard' (multivalued) for 'implicit' route

          One should be able to create a collection w/ no shards also (although normally the next steps would be to create shards).

          Show
          Yonik Seeley added a comment - 'numShards is a required param for default 'compositeId' router. We want things to be as easy as possible to get going... it may make sense to have a default of 1 for numShards. The name of the Router is a bit confusing. I fee it should be called "explicit" instead of "implicit". But let us leave it Yeah... I guess what I was thinking at the time was when someone sent and update to a shard w/ no further routing info. The update is assumed to belong on the receiving shard (that's the implicit part). Of course it should work with explict shard info too. new required parameter 'shard' (multivalued) for 'implicit' route One should be able to create a collection w/ no shards also (although normally the next steps would be to create shards).
          Hide
          Noble Paul added a comment -

          Assigning 'implicit' router in the absence of 'numShards' param is confusing for the user. I would like it this way. there is a param caller 'router' whose default is "compositeId" . if both 'numShards' and 'router' params are absent the create action should fail stating 'numShards is a required param for default 'compositeId' router.

          Proposed modifications to the collection CREATE action

          • creating a shard should be possible even if no nodes can be allocated to the shard. As and when nodes come up , it can be allocated to the shard
          • new required parameter 'shard' (multivalued) for 'implicit' route

          New collection action , CREATE_SHARD

          • params . 'shard' (multivalued) optional . It creates new shards and try to allocate nodes for the same. If enough nodes are not available the shards will remain in 'construction' state
          • replicationFactor : (optional) if absent the value will be inherited from the collection

          New collection action , UPDATE_SHARD

          • replicationFactor : provide a new value to add capacity to an existing shard

          The name of the Router is a bit confusing. I fee it should be called "explicit" instead of "implicit". But let us leave it

          Show
          Noble Paul added a comment - Assigning 'implicit' router in the absence of 'numShards' param is confusing for the user. I would like it this way. there is a param caller 'router' whose default is "compositeId" . if both 'numShards' and 'router' params are absent the create action should fail stating 'numShards is a required param for default 'compositeId' router. Proposed modifications to the collection CREATE action creating a shard should be possible even if no nodes can be allocated to the shard. As and when nodes come up , it can be allocated to the shard new required parameter 'shard' (multivalued) for 'implicit' route New collection action , CREATE_SHARD params . 'shard' (multivalued) optional . It creates new shards and try to allocate nodes for the same. If enough nodes are not available the shards will remain in 'construction' state replicationFactor : (optional) if absent the value will be inherited from the collection New collection action , UPDATE_SHARD replicationFactor : provide a new value to add capacity to an existing shard The name of the Router is a bit confusing. I fee it should be called "explicit" instead of "implicit". But let us leave it
          Hide
          Noble Paul added a comment - - edited

          when creating a collection, the ability to specify a router (and associated config for the router) and have that persisted, etc

          I guess this can be a separate issue and can be fixed right away

          using request parameters or document fields to direct what shard an update is for

          Is there a DocRouter which shards on the basis of document fields? (Other than the composite Id router) . I feel we need another one which does not change id of the document

          Show
          Noble Paul added a comment - - edited when creating a collection, the ability to specify a router (and associated config for the router) and have that persisted, etc I guess this can be a separate issue and can be fixed right away using request parameters or document fields to direct what shard an update is for Is there a DocRouter which shards on the basis of document fields? (Other than the composite Id router) . I feel we need another one which does not change id of the document
          Hide
          Yonik Seeley added a comment - - edited

          Off the top of my head, some things left to do in this general area of "custom sharding":

          • ability to create a new shard via collections API (and other lifecycle operations... delete, rename?)
            • probably want to have the option of either leaving it empty, or creating new replicas
          • anything that may make time based sharding easier
            • this probably includes shard aliasing, like we have collection aliasing now
          • when creating a collection, the ability to specify a router (and associated config for the router) and have that persisted, etc
          • using request parameters or document fields to direct what shard an update is for
            • I think a lot of this is done/working (implicit router), but lacks tests

          Further down the road, we also need to consider what to do about deletes, real-time get, or other operations where we would normally derive the shard from the ID... if it's not provided in another form, then we could decide to broadcast to all shards.

          Show
          Yonik Seeley added a comment - - edited Off the top of my head, some things left to do in this general area of "custom sharding": ability to create a new shard via collections API (and other lifecycle operations... delete, rename?) probably want to have the option of either leaving it empty, or creating new replicas anything that may make time based sharding easier this probably includes shard aliasing, like we have collection aliasing now when creating a collection, the ability to specify a router (and associated config for the router) and have that persisted, etc using request parameters or document fields to direct what shard an update is for I think a lot of this is done/working (implicit router), but lacks tests Further down the road, we also need to consider what to do about deletes, real-time get, or other operations where we would normally derive the shard from the ID... if it's not provided in another form, then we could decide to broadcast to all shards.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Yonik Seeley
          http://svn.apache.org/viewvc?view=revision&revision=1424265

          SOLR-4221: pick correct router for collection props

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Yonik Seeley http://svn.apache.org/viewvc?view=revision&revision=1424265 SOLR-4221 : pick correct router for collection props
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Yonik Seeley
          http://svn.apache.org/viewvc?view=revision&revision=1424263

          SOLR-4221: pick correct router for collection props

          Show
          Commit Tag Bot added a comment - [trunk commit] Yonik Seeley http://svn.apache.org/viewvc?view=revision&revision=1424263 SOLR-4221 : pick correct router for collection props

            People

            • Assignee:
              Noble Paul
              Reporter:
              Yonik Seeley
            • Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development