Solr
  1. Solr
  2. SOLR-2820

Add both model and state to ZooKeeper layout for SolrCloud

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Current we skimp by here by having the model and simple node state - we really want the model and full cluster state longer term though.

        Issue Links

          Activity

          Hide
          Ted Dunning added a comment -

          Would it help to have a toy implementation for discussion here? I don't have enough time to make clean updates to Solr itself, but I have built this kind of code several times and could build a simple framework very quickly.

          Show
          Ted Dunning added a comment - Would it help to have a toy implementation for discussion here? I don't have enough time to make clean updates to Solr itself, but I have built this kind of code several times and could build a simple framework very quickly.
          Hide
          Mark Miller added a comment -

          Sure, I wouldn't say no. An small impl to compare with and steal from would be great...

          Show
          Mark Miller added a comment - Sure, I wouldn't say no. An small impl to compare with and steal from would be great...
          Hide
          Jamie Johnson added a comment -

          Is this task to update /cluster_state with the state of each individual replica? Would this just be having the shard leader watch /live_nodes and on change check update the ClusterState so that particular replica will be marked as down and having some mechanism by which the replicas state could be changed (through some admin command) and again update the ClusterState?

          Show
          Jamie Johnson added a comment - Is this task to update /cluster_state with the state of each individual replica? Would this just be having the shard leader watch /live_nodes and on change check update the ClusterState so that particular replica will be marked as down and having some mechanism by which the replicas state could be changed (through some admin command) and again update the ClusterState?
          Hide
          Ted Dunning added a comment -

          Both alternatives would work. My own tendency would be for nodes to update state themselves to avoid round-trips to ZK.

          Show
          Ted Dunning added a comment - Both alternatives would work. My own tendency would be for nodes to update state themselves to avoid round-trips to ZK.
          Hide
          Mark Miller added a comment -

          My initial thoughts are (I've got to go re read some of Ted's comments):

          We keep track of the target layout and the current layout. The target layout would include things like nodes that are down - you would be able to tell a slice should exist even if all nodes serving that slice where down. The current layout would show how things actually are - what is up, what is down, etc.

          I suppose you might track this all in one structure, but we want to think about what the separation might give us too.

          For example, suppose we offer manual rebalancing control (supposing getting the heuristics right for auto is hard difficult or not always appropriate) - if a user where to move a shard to another node, he could make the change in the target layout. If the overseer goes down while making this change, the new overseer will be able to look and see that while a change was intended, it has not happened yet and the shard still needs to be moved. Meanwhile, clients will still be directed to the old serving node through the current layout.

          On the other hand, clients will still have to read both current and target layouts to be aware of missing slices and properly handle partial results. It still does not give you "one place for all of this" that I remember Ted arguing for.

          Show
          Mark Miller added a comment - My initial thoughts are (I've got to go re read some of Ted's comments): We keep track of the target layout and the current layout. The target layout would include things like nodes that are down - you would be able to tell a slice should exist even if all nodes serving that slice where down. The current layout would show how things actually are - what is up, what is down, etc. I suppose you might track this all in one structure, but we want to think about what the separation might give us too. For example, suppose we offer manual rebalancing control (supposing getting the heuristics right for auto is hard difficult or not always appropriate) - if a user where to move a shard to another node, he could make the change in the target layout. If the overseer goes down while making this change, the new overseer will be able to look and see that while a change was intended, it has not happened yet and the shard still needs to be moved. Meanwhile, clients will still be directed to the old serving node through the current layout. On the other hand, clients will still have to read both current and target layouts to be aware of missing slices and properly handle partial results. It still does not give you "one place for all of this" that I remember Ted arguing for.
          Hide
          Jamie Johnson added a comment -

          Ok, makes sense. What is the process for performing a state change to a replica? I am not super familiar with how this would happen, would another method need to be added to ZkController to handle this? Also how do we handle nodes that just die? In that case they won't be able to update their state so we'll need some overseer to do this right? Would the shard leader be the appropriate place?

          Show
          Jamie Johnson added a comment - Ok, makes sense. What is the process for performing a state change to a replica? I am not super familiar with how this would happen, would another method need to be added to ZkController to handle this? Also how do we handle nodes that just die? In that case they won't be able to update their state so we'll need some overseer to do this right? Would the shard leader be the appropriate place?
          Hide
          Mark Miller added a comment -

          Yeah, I think we would need an overseer for this? Probably not the shard leader, because it would also have to update the state when a whole "slice" goes down.

          I'm waiting to see what Ted is working on.

          Show
          Mark Miller added a comment - Yeah, I think we would need an overseer for this? Probably not the shard leader, because it would also have to update the state when a whole "slice" goes down. I'm waiting to see what Ted is working on.

            People

            • Assignee:
              Unassigned
              Reporter:
              Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development