Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1515

Long reconnect timeout if leader failed.

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.5
    • Fix Version/s: None
    • Component/s: leaderElection, quorum, server
    • Labels:
    • Environment:

      Gentoo linux, but every environment is affected.

      Description

      In zookeeper 3.3.5 in file src/java/main/org/apache/zookeeper/server/quorum/Learner.java:325 you may see Thread.sleep(1000);

      This is always happens after leader failure or restart. Zookeeper reelects new leader and all followers try to connect to it. But first attempt always fails because of "Connection refused":

      2012-07-23 18:55:48,159 - WARN [QuorumPeer:/0.0.0.0:2181:Learner@229] - Unexpected exception, tries=0, connecting to web329.local/192.168.1.74:2888
      java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:529)
      at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:221)
      at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:65)
      at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:645)

      I propose to change this line to the next code:

      Learner.java
      if (tries > 0) {
          Thread.sleep(self.tickTime);
      }
      

      This way first reconnect attempt will be done immediately, other will wait for tick time (this is good semantic change, I suppose).

      The result of this change - leader reelection time lowered from >1500ms to 300-400ms with 50ms tick time. This is pretty important for our production environment and will not break any existing installations.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bobrik Ivan Babrou
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: