[KAFKA-1903] Zk Expiration causes controller deadlock

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 0.8.1, 0.8.1.1
• Component/s: controller
• Environment: java version "1.7.0_55"; Java(TM) SE Runtime Environment (build 1.7.0_55-b13); Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode); ; kafka_2.9.2-0.8.1
• Assignee: Neha Narkhede
• Reporter: yufeng.chen
• Created: 28/Jan/15 08:05
• Updated: 07/Feb/15 20:02
• Resolved: 29/Jan/15 14:28

Description

when controller encounter a ZK expired， zookeeper node /broker/ids lost one kafkk controler. If there has three node, e.g. 1 2, 3; and the 1 start delete-topic-method thread. At this time, node 1 will lost. Why? The reason is that: when ZK expiration happened, the zk-event-thread will call KafkaController.SessionExpirationListener.handleNewSession method. if the zk-event-thread has the controllerContext.controllerLock, will call onControllerResignation->deleteTopicManager.shutdown()>deleteTopicsThread.shutdown(). And the delete-topic-thread is working, and await at awaitTopicDeletionNotification() method。 Zk-event-thread call deleteTopicsThread.shutdown() and wait until the run() method execute compelely. Because the zk-event-thread has the lock, "deleteTopicsCond.await()" whill not be really "interruted " . Then zk-event-thread whill pause, not execute the kafkaHealthcheck>SessionExpireListener.handleNewSession。 The controller will not register again. The jstack log :

"delete-topics-thread" prio=10 tid=0x00007fb0bc21b000 nid=0x2825 waiting on condition [0x00007fb0f534a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000e4952da0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047)
  at kafka.controller.TopicDeletionManager.kafka$controller$TopicDeletionManager$$awaitTopicDeletionNotification(TopicDeletionManager.scala:178)
  at kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply$mcV$sp(TopicDeletionManager.scala:334)
  at kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:333)
  at kafka.controller.TopicDeletionManager$DeleteTopicsThread$$anonfun$doWork$1.apply(TopicDeletionManager.scala:333)
  at kafka.utils.Utils$.inLock(Utils.scala:538)
  at kafka.controller.TopicDeletionManager$DeleteTopicsThread.doWork(TopicDeletionManager.scala:333)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
  "ZkClient-EventThread-12-10.3.63.8:2181,10.3.63.9:2181" daemon prio=10 tid=0x00007fb10038e800 nid=0x7d93 waiting on condition [0x00007fb0f544a000]
  java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
• parking to wait for <0x00000000e4f4a760> (a java.util.concurrent.CountDownLatch$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
  at kafka.utils.ShutdownableThread.shutdown(ShutdownableThread.scala:36)
  at kafka.controller.TopicDeletionManager.shutdown(TopicDeletionManager.scala:93)
  at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:340)
  at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
  at kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
  at kafka.utils.Utils$.inLock(Utils.scala:538)
  at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:337)
  at kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1068)
  at kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
  at kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
  at kafka.utils.Utils$.inLock(Utils.scala:538)
  at kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1067)
  at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
  at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)