Solr
  1. Solr
  2. SOLR-6763

Shard leader election thread can persist across session expiry

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10.4, 5.0
    • Component/s: None
    • Labels:
      None

      Description

      A ZK connection loss during a call to ElectionContext.waitForReplicasToComeUp() will result in two leader election processes for the shard running within a single node - the initial election that was waiting, and another spawned by the ReconnectStrategy. After the function returns, the first election will create an ephemeral leader node. The second election will then also attempt to create this node, fail, and try to put itself into recovery. It will also set the 'isLeader' value in its CloudDescriptor to false.

      The first election, meanwhile, is happily maintaining the ephemeral leader node. But any updates that are sent to the shard will cause an exception due to the mismatch between the cloudstate (where this node is the leader) and the local CloudDescriptor leader state.

      I think the fix is straightfoward - the call to zkClient.getChildren() in waitForReplicasToComeUp should be called with 'retryOnReconnect=false', rather than 'true' as it is currently, because once the connection has dropped we're going to launch a new election process anyway.

      1. SOLR-6763.patch
        3 kB
        Alan Woodward
      2. SOLR-6763.patch
        2 kB
        Alan Woodward

        Activity

        Hide
        Mark Miller added a comment -

        and another spawned by the ReconnectStrategy.

        Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.

        Show
        Mark Miller added a comment - and another spawned by the ReconnectStrategy. Hmm...this sounds fishy. We should not be spawning any new election thread on ConnectionLoss - only on Expiration.
        Hide
        Mark Miller added a comment -

        Which version did you see this on by the way?

        Show
        Mark Miller added a comment - Which version did you see this on by the way?
        Hide
        Alan Woodward added a comment -

        This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).

        Show
        Alan Woodward added a comment - This is on 5.x. And you're right, it was actually caused by session expiry, not connection loss (runaway query caused a massive GC pause).
        Hide
        Mark Miller added a comment -

        Hmm...have to look closer then, but in that case the fix doesn't sound right.

        Show
        Mark Miller added a comment - Hmm...have to look closer then, but in that case the fix doesn't sound right.
        Hide
        Alan Woodward added a comment -

        Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.

        Show
        Alan Woodward added a comment - Yeah, I think the important thing to do here is to bail out on a SessionExpiredException. So the added try-catch clause in the above patch will fix it, but we want to keep the getChildren() call with retryOnReconnect=true.
        Hide
        Alan Woodward added a comment -

        Patch with the better fix.

        Show
        Alan Woodward added a comment - Patch with the better fix.
        Hide
        Mark Miller added a comment -

        +1, good catch!

        Show
        Mark Miller added a comment - +1, good catch!
        Hide
        ASF subversion and git services added a comment -

        Commit 1641590 from Alan Woodward in branch 'dev/trunk'
        [ https://svn.apache.org/r1641590 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        ASF subversion and git services added a comment - Commit 1641590 from Alan Woodward in branch 'dev/trunk' [ https://svn.apache.org/r1641590 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        ASF subversion and git services added a comment -

        Commit 1641592 from Alan Woodward in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1641592 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        ASF subversion and git services added a comment - Commit 1641592 from Alan Woodward in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1641592 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        Alan Woodward added a comment -

        Thanks for the review Mark!

        Show
        Alan Woodward added a comment - Thanks for the review Mark!
        Hide
        Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        Anshum Gupta added a comment - Bulk close after 5.0 release.
        Hide
        Shalin Shekhar Mangar added a comment -

        Reopening to backport to 4.10.4

        Show
        Shalin Shekhar Mangar added a comment - Reopening to backport to 4.10.4
        Hide
        ASF subversion and git services added a comment -

        Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1662432 ]

        SOLR-6763: Shard leader elections should not persist across ZK session expiry

        Show
        ASF subversion and git services added a comment - Commit 1662432 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1662432 ] SOLR-6763 : Shard leader elections should not persist across ZK session expiry
        Hide
        Michael McCandless added a comment -

        Bulk close for 4.10.4 release

        Show
        Michael McCandless added a comment - Bulk close for 4.10.4 release

          People

          • Assignee:
            Alan Woodward
            Reporter:
            Alan Woodward
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development