Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-4220

Potential redundant connection attempts during leader election

    XMLWordPrintableJSON

    Details

      Description

      We've seen a few failures or long delays in electing a new leader when the previous one has a hard host reset (as opposed to just the service process down, since connections don't need to wait for timeout there). Symptoms are similar to https://issues.apache.org/jira/browse/ZOOKEEPER-2164. Reducing cnxTimeout from 5 to 1.5 seconds makes the problem much less frequent, but doesn't fix it completely. We are still using an old ZooKeeper version (3.5.5), and the new async connect feature will presumably avoid it.

      But we noticed a pattern of twice the expected number of connection attempts to the same downed instance in the log, and it appears to be due to a code glitch in QuorumCnxManager.java:

       

      synchronized void connectOne(long sid) {
          ...
          if (lastCommittedView.containsKey(sid)) {
              knownId = true;
              if (connectOne(sid, lastCommittedView.get(sid).electionAddr))
                  return;
          }
          if (lastSeenQV != null && lastProposedView.containsKey(sid)
                  && (!knownId || (lastProposedView.get(sid).electionAddr !=   <----
                  lastCommittedView.get(sid).electionAddr))) {
              knownId = true;
              if (connectOne(sid, lastProposedView.get(sid).electionAddr))
                  return;
          }
      

      Comparing electionAddrs should be done with !equals presumably, otherwise connectOne will be invoked an extra time even in the common case when the addresses do match.

      The code around it has changed recently, but the check itself still exists at the top of master. It might not matter as much with the async connects, but perhaps it helps even then.

        Attachments

          Activity

            People

            • Assignee:
              symat Mate Szalay-Beko
              Reporter:
              mirg Alex Mirgorodskiy
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 50m
                3h 50m