Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6879

Controller deadlock following session expiration

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.1, 2.0.0
    • Component/s: controller
    • Labels:
      None

      Description

      We have observed an apparent deadlock situation which occurs following a session expiration. The suspected deadlock occurs between the zookeeper "initializationLock" and the latch inside the Expire event which we use to ensure all events have been handled.

      In the logs, we see the "Session expired" message following acquisition of the initialization lock: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L358

      But we never see any logs indicating that the new session is being initialized. In fact, the controller logs are basically empty from that point on. The problem we suspect is that completion of the beforeInitializingSession callback requires that all events have finished processing in order to count down the latch: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala#L1525.

      But an event which was dequeued just prior to the acquisition of the write lock may be unable to complete because it is awaiting acquisition of the initialization lock: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L137.

      The impact is that the broker continues in a zombie state. It continues fetching and is periodically added to ISRs, but it never receives any further requests from the controller since it is not registered.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: