Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6763

Shard leader election thread can persist across session expiry

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.10.4, 5.0
    • None
    • None

    Description

      A ZK connection loss during a call to ElectionContext.waitForReplicasToComeUp() will result in two leader election processes for the shard running within a single node - the initial election that was waiting, and another spawned by the ReconnectStrategy. After the function returns, the first election will create an ephemeral leader node. The second election will then also attempt to create this node, fail, and try to put itself into recovery. It will also set the 'isLeader' value in its CloudDescriptor to false.

      The first election, meanwhile, is happily maintaining the ephemeral leader node. But any updates that are sent to the shard will cause an exception due to the mismatch between the cloudstate (where this node is the leader) and the local CloudDescriptor leader state.

      I think the fix is straightfoward - the call to zkClient.getChildren() in waitForReplicasToComeUp should be called with 'retryOnReconnect=false', rather than 'true' as it is currently, because once the connection has dropped we're going to launch a new election process anyway.

      Attachments

        1. SOLR-6763.patch
          3 kB
          Alan Woodward
        2. SOLR-6763.patch
          2 kB
          Alan Woodward

        Activity

          markrmiller@gmail.com Mark Miller added a comment -

          and another spawned by the ReconnectStrategy.

          Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.

          markrmiller@gmail.com Mark Miller added a comment - and another spawned by the ReconnectStrategy. Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.
          markrmiller@gmail.com Mark Miller added a comment -

          Which version did you see this on by the way?

          markrmiller@gmail.com Mark Miller added a comment - Which version did you see this on by the way?
          romseygeek Alan Woodward added a comment -

          This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).

          romseygeek Alan Woodward added a comment - This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).
          markrmiller@gmail.com Mark Miller added a comment -

          Hmm...have to look closer then, but in that case the fix doesn't sound right.

          markrmiller@gmail.com Mark Miller added a comment - Hmm...have to look closer then, but in that case the fix doesn't sound right.
          romseygeek Alan Woodward added a comment -

          Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.

          romseygeek Alan Woodward added a comment - Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.
          romseygeek Alan Woodward added a comment -

          Patch with the better fix.

          romseygeek Alan Woodward added a comment - Patch with the better fix.
          markrmiller@gmail.com Mark Miller added a comment -

          +1, good catch!

          markrmiller@gmail.com Mark Miller added a comment - +1, good catch!

          Commit 1641590 from romseygeek in branch 'dev/trunk'
          [ https://svn.apache.org/r1641590 ]

          SOLR-6763: Shard leader elections should not persist across ZK session expiry

          jira-bot ASF subversion and git services added a comment - Commit 1641590 from romseygeek in branch 'dev/trunk' [ https://svn.apache.org/r1641590 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry

          Commit 1641592 from romseygeek in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1641592 ]

          SOLR-6763: Shard leader elections should not persist across ZK session expiry

          jira-bot ASF subversion and git services added a comment - Commit 1641592 from romseygeek in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1641592 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
          romseygeek Alan Woodward added a comment -

          Thanks for the review Mark!

          romseygeek Alan Woodward added a comment - Thanks for the review Mark!
          anshum Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          anshum Anshum Gupta added a comment - Bulk close after 5.0 release.

          Reopening to backport to 4.10.4

          shalin Shalin Shekhar Mangar added a comment - Reopening to backport to 4.10.4

          Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10'
          [ https://svn.apache.org/r1662432 ]

          SOLR-6763: Shard leader elections should not persist across ZK session expiry

          jira-bot ASF subversion and git services added a comment - Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1662432 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry

          Bulk close for 4.10.4 release

          mikemccand Michael McCandless added a comment - Bulk close for 4.10.4 release

          People

            romseygeek Alan Woodward
            romseygeek Alan Woodward
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: