Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
1.1.0
-
None
Description
We have observed an apparent deadlock situation which occurs following a session expiration. The suspected deadlock occurs between the zookeeper "initializationLock" and the latch inside the Expire event which we use to ensure all events have been handled.
In the logs, we see the "Session expired" message following acquisition of the initialization lock: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L358
But we never see any logs indicating that the new session is being initialized. In fact, the controller logs are basically empty from that point on. The problem we suspect is that completion of the beforeInitializingSession callback requires that all events have finished processing in order to count down the latch: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala#L1525.
But an event which was dequeued just prior to the acquisition of the write lock may be unable to complete because it is awaiting acquisition of the initialization lock: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L137.
The impact is that the broker continues in a zombie state. It continues fetching and is periodically added to ISRs, but it never receives any further requests from the controller since it is not registered.
Attachments
Issue Links
- links to