Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4229

Controller can't start after several zk expired event

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
    • Fix Version/s: 0.10.2.0
    • Component/s: controller
    • Labels:

      Description

      We found the controller not started after several zk expired event in our test environment. By analysing the log, I found the controller will handle the ephemeral node data delete event first and then the zk expired event , then the controller will gone.
      I can reproducer it on my develop env:
      1. set up a one broker and one zk env, specify a very large zk timeout (20s)
      2. stop the broker and remove the zk's /broker/ids/0 directory
      3. restart the broker and make a breakpoint in the zk client's event thread to queue the delete event.
      4. after the /controller node gone the breakpoint will hit.
      5. expired the current session(suspend the send thread) and create a new session s2
      6. resume the event thread, then the controller will handle LeaderChangeListener.handleDataDeleted and become leader
      7. then controller will handle SessionExpirationListener.handleNewSession, it resign the controller and elect, but when elect it found the /controller node is exist and not become the leader. But the /controller node is created by current session s2 will not remove. So the controller is gone

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pengwei Pengwei
                Reporter:
                pengwei Pengwei
                Reviewer:
                Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: