Solr
  1. Solr
  2. SOLR-3750

On session expiration, we should explicitly wait some time before running the leader sync process so that we are sure every node participates.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      We should wait until all the known nodes are part of the election, or X amount of time if that does not happen (when a node or more does not come back for whatever reason).

        Issue Links

          Activity

          Hide
          Mark Miller added a comment -

          I think initially we should wait for the known amount of replicas to show up, else a set amount of time (N).

          We need something like this for a start, but it seems it could be improved. The ugly case seems to be when 2 or more nodes go down and are not coming back anytime soon...updates would then be down for N amount of time.

          Show
          Mark Miller added a comment - I think initially we should wait for the known amount of replicas to show up, else a set amount of time (N). We need something like this for a start, but it seems it could be improved. The ugly case seems to be when 2 or more nodes go down and are not coming back anytime soon...updates would then be down for N amount of time.
          Hide
          Mark Miller added a comment -

          To clarify, the above comment is more in regards to the leader goes down case. In the connectionloss case, usually everyone should come right back in the normal case. When a leader goes down, we can look for numReplicas-1 - but if 2 or more nodes go down, we would have to wait N amount of time.

          Show
          Mark Miller added a comment - To clarify, the above comment is more in regards to the leader goes down case. In the connectionloss case, usually everyone should come right back in the normal case. When a leader goes down, we can look for numReplicas-1 - but if 2 or more nodes go down, we would have to wait N amount of time.
          Hide
          Mark Miller added a comment -

          I'd like to come up with a goo N to wait - initially I'm going with 5 min. Suggestions?

          Show
          Mark Miller added a comment - I'd like to come up with a goo N to wait - initially I'm going with 5 min. Suggestions?
          Hide
          Mark Miller added a comment -

          I suppose one way to kick the can is to make this configurable - which % of replicas to wait and see up before leader election, and how long to wait for that many to show up.

          Show
          Mark Miller added a comment - I suppose one way to kick the can is to make this configurable - which % of replicas to wait and see up before leader election, and how long to wait for that many to show up.
          Hide
          Mark Miller added a comment -

          To add further - I don't think this is strictly necessary right now - due to the fact that we require updates to hit all replicas before returning success or failure. It will become a nice option once we let you be less strict with that though.

          Show
          Mark Miller added a comment - To add further - I don't think this is strictly necessary right now - due to the fact that we require updates to hit all replicas before returning success or failure. It will become a nice option once we let you be less strict with that though.
          Hide
          Mark Miller added a comment -

          Because this behavior is not currently strictly necessary, and sometimes not ideal, I'm thinking I will default it to false and allow it to be configured to on along with the amount of time to wait - at least initially.

          Later, when we allow you to return before every replica has an update, this could probably be automatically enabled? Or in the case you could choose that behavior per update, doc'd that you should enable it.

          I think it should be an option now because it also can offer some protections against more esoteric cases - eg when starting the cluster, you start using a really old node and start that node first and it becomes leader...you just want to be willing to live with some of the esoteric downsides of enabling it as as well.

          Show
          Mark Miller added a comment - Because this behavior is not currently strictly necessary, and sometimes not ideal, I'm thinking I will default it to false and allow it to be configured to on along with the amount of time to wait - at least initially. Later, when we allow you to return before every replica has an update, this could probably be automatically enabled? Or in the case you could choose that behavior per update, doc'd that you should enable it. I think it should be an option now because it also can offer some protections against more esoteric cases - eg when starting the cluster, you start using a really old node and start that node first and it becomes leader...you just want to be willing to live with some of the esoteric downsides of enabling it as as well.
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development