ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-822

Leader election taking a long time to complete

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.2, 3.4.0
    • Component/s: quorum
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Created a 3 node cluster.

      1 Fail the ZK leader
      2. Let leader election finish. Restart the leader and let it join the
      3. Repeat

      After a few rounds leader election takes anywhere 25- 60 seconds to finish. Note- we didn't have any ZK clients and no new znodes were created.

      zoo.cfg is shown below:

      #Mon Jul 19 12:15:10 UTC 2010
      server.1=192.168.4.12\:2888\:3888
      server.0=192.168.4.11\:2888\:3888
      clientPort=2181
      dataDir=/var/zookeeper
      syncLimit=2
      server.2=192.168.4.13\:2888\:3888
      initLimit=5
      tickTime=2000

      I have attached logs from two nodes that took a long time to form the cluster after failing the leader. The leader was down anyways so logs from that node shouldn't matter.
      Look for "START HERE". Logs after that point should be of our interest.

      1. ZOOKEEPER-822-3.3.2.patch
        7 kB
        Flavio Junqueira
      2. ZOOKEEPER-822-3.3.2.patch
        7 kB
        Flavio Junqueira
      3. ZOOKEEPER-822-3.3.2.patch
        8 kB
        Flavio Junqueira
      4. ZOOKEEPER-822-3.3.2.patch
        9 kB
        Flavio Junqueira
      5. ZOOKEEPER-822.patch_v1
        6 kB
        Vishal Kher
      6. ZOOKEEPER-822.patch
        5 kB
        Flavio Junqueira
      7. ZOOKEEPER-822.patch
        7 kB
        Flavio Junqueira
      8. ZOOKEEPER-822.patch
        7 kB
        Flavio Junqueira
      9. ZOOKEEPER-822.patch
        7 kB
        Flavio Junqueira
      10. ZOOKEEPER-822.patch
        8 kB
        Flavio Junqueira
      11. ZOOKEEPER-822.patch
        9 kB
        Flavio Junqueira
      12. zookeeper-3.4.0.tar.gz
        33 kB
        Vishal Kher
      13. zk_leader_election.tar.gz
        53 kB
        Vishal Kher
      14. test_zookeeper_2.log
        215 kB
        Vishal Kher
      15. test_zookeeper_1.log
        212 kB
        Vishal Kher
      16. rhel.tar.gz
        91 kB
        Vishal Kher
      17. 822.tar.gz
        47 kB
        Vishal Kher

        Activity

        Vishal Kher created issue -
        Vishal Kher made changes -
        Field Original Value New Value
        Attachment test_zookeeper_2.log [ 12449852 ]
        Attachment test_zookeeper_1.log [ 12449853 ]
        Vishal Kher made changes -
        Attachment zk_leader_election.tar.gz [ 12450233 ]
        Vishal Kher made changes -
        Attachment 822.tar.gz [ 12451046 ]
        Vishal Kher made changes -
        Attachment rhel.tar.gz [ 12452439 ]
        Vishal Kher made changes -
        Attachment zookeeper-3.4.0.tar.gz [ 12452676 ]
        Mahadev konar made changes -
        Assignee Vishal K [ vishalmlst ]
        Fix Version/s 3.3.2 [ 12315108 ]
        Fix Version/s 3.4.0 [ 12314469 ]
        Vishal Kher made changes -
        Attachment ZOOKEEPER-822.patch_v1 [ 12454017 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12454641 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12454886 ]
        Flavio Junqueira made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822-3.3.2.patch [ 12454889 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12454890 ]
        Flavio Junqueira made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822-3.3.2.patch [ 12455045 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12455046 ]
        Flavio Junqueira made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Flavio Junqueira made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822-3.3.2.patch [ 12455607 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12455608 ]
        Flavio Junqueira made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Flavio Junqueira made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822-3.3.2.patch [ 12455854 ]
        Flavio Junqueira made changes -
        Attachment ZOOKEEPER-822.patch [ 12455855 ]
        Flavio Junqueira made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Benjamin Reed made changes -
        Hadoop Flags [Reviewed]
        Patrick Hunt made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Mahadev konar made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Vishal Kher
            Reporter:
            Vishal Kher
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development