Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6763

Shard leader election thread can persist across session expiry

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10.4, 5.0
    • Component/s: None
    • Labels:
      None

      Description

      A ZK connection loss during a call to ElectionContext.waitForReplicasToComeUp() will result in two leader election processes for the shard running within a single node - the initial election that was waiting, and another spawned by the ReconnectStrategy. After the function returns, the first election will create an ephemeral leader node. The second election will then also attempt to create this node, fail, and try to put itself into recovery. It will also set the 'isLeader' value in its CloudDescriptor to false.

      The first election, meanwhile, is happily maintaining the ephemeral leader node. But any updates that are sent to the shard will cause an exception due to the mismatch between the cloudstate (where this node is the leader) and the local CloudDescriptor leader state.

      I think the fix is straightfoward - the call to zkClient.getChildren() in waitForReplicasToComeUp should be called with 'retryOnReconnect=false', rather than 'true' as it is currently, because once the connection has dropped we're going to launch a new election process anyway.

      1. SOLR-6763.patch
        2 kB
        Alan Woodward
      2. SOLR-6763.patch
        3 kB
        Alan Woodward

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Bulk close for 4.10.4 release

        Show
        mikemccand Michael McCandless added a comment - Bulk close for 4.10.4 release
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1662432 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1662432 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Reopening to backport to 4.10.4

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Reopening to backport to 4.10.4
        Hide
        anshumg Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        anshumg Anshum Gupta added a comment - Bulk close after 5.0 release.
        Hide
        romseygeek Alan Woodward added a comment -

        Thanks for the review Mark!

        Show
        romseygeek Alan Woodward added a comment - Thanks for the review Mark!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1641592 from Alan Woodward in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1641592 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1641592 from Alan Woodward in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1641592 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1641590 from Alan Woodward in branch 'dev/trunk'
        [ https://svn.apache.org/r1641590 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1641590 from Alan Woodward in branch 'dev/trunk' [ https://svn.apache.org/r1641590 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        +1, good catch!

        Show
        markrmiller@gmail.com Mark Miller added a comment - +1, good catch!
        Hide
        romseygeek Alan Woodward added a comment -

        Patch with the better fix.

        Show
        romseygeek Alan Woodward added a comment - Patch with the better fix.
        Hide
        romseygeek Alan Woodward added a comment -

        Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.

        Show
        romseygeek Alan Woodward added a comment - Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Hmm...have to look closer then, but in that case the fix doesn't sound right.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Hmm...have to look closer then, but in that case the fix doesn't sound right.
        Hide
        romseygeek Alan Woodward added a comment -

        This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).

        Show
        romseygeek Alan Woodward added a comment - This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Which version did you see this on by the way?

        Show
        markrmiller@gmail.com Mark Miller added a comment - Which version did you see this on by the way?
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        and another spawned by the ReconnectStrategy.

        Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.

        Show
        markrmiller@gmail.com Mark Miller added a comment - and another spawned by the ReconnectStrategy. Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.

          People

          • Assignee:
            romseygeek Alan Woodward
            Reporter:
            romseygeek Alan Woodward
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development