Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4409

ZK consumer shutdown/topic event deadlock

Agile BoardAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.1.0
    • 0.10.1.1
    • consumer, core
    • None

    Description

      This only applies to the old zookeeper consumer. It is trivial enough to fix.

      The consumer can deadlock on shutdown if a topic event fires during shutdown. The shutdown acquires the rebalance lock and then the topic-event-watcher lock. The topic event watcher acquires these in the reverse order. Shutdown should not need to acquire the topic-event-watcher’s lock - all it does is unsubscribes from topic events.

      Stack trace:

      "mirrormaker-thread-0":
              at kafka.consumer.ZookeeperTopicEventWatcher.shutdown(ZookeeperTopicEventWatcher.scala:50)
              - waiting to lock <0x000000072a65d508> (a java.lang.Object)
              at kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:216)
              - locked <0x00000007103c69c0> (a java.lang.Object)
              at kafka.tools.MirrorMaker$MirrorMakerOldConsumer.cleanup(MirrorMaker.scala:519)
              at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply$mcV$sp(MirrorMaker.scala:441)
              at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:76)
              at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
              at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:47)
              at kafka.utils.Logging$class.swallow(Logging.scala:94)
              at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:47)
              at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:441)
      "ZkClient-EventThread-58-<zkconnect>":
              at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:639)
              - waiting to lock <0x00000007103c69c0> (a java.lang.Object)
              at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:982)
              at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.handleTopicEvent(ZookeeperConsumerConnector.scala:1048)
              at kafka.consumer.ZookeeperTopicEventWatcher$ZkTopicEventListener.liftedTree1$1(ZookeeperTopicEventWatcher.scala:69)
              at kafka.consumer.ZookeeperTopicEventWatcher$ZkTopicEventListener.handleChildChange(ZookeeperTopicEventWatcher.scala:65)
              - locked <0x000000072a65d508> (a java.lang.Object)
              at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)
              at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
      Found one Java-level deadlock:
      =============================
      "mirrormaker-thread-0":
        waiting to lock monitor 0x00007f1f38029748 (object 0x000000072a65d508, a java.lang.Object),
        which is held by "ZkClient-EventThread-58-<zkconnect>"
      "ZkClient-EventThread-58-<zkconnect>":
        waiting to lock monitor 0x00007f1e900249a8 (object 0x00000007103c69c0, a java.lang.Object),
        which is held by "mirrormaker-thread-0"
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jjkoshy Joel Jacob Koshy
            jjkoshy Joel Jacob Koshy
            Jiangjie Qin Jiangjie Qin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment