Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1724

Errors after reboot in single node setup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.0
    • Fix Version/s: 0.9.0.0
    • Component/s: None
    • Labels:

      Description

      In a single node setup, after reboot, Kafka logs show the following:

      [2014-10-22 16:37:22,206] INFO [Controller 0]: Controller starting up (kafka.controller.KafkaController)
      [2014-10-22 16:37:22,419] INFO [Controller 0]: Controller startup complete (kafka.controller.KafkaController)
      [2014-10-22 16:37:22,554] INFO conflict in /brokers/ids/0 data: {"jmx_port":-1,"timestamp":"1413995842465","host":"ip-10-91-142-54.eu-west-1.compute.internal","version":1,"port":9092} stored data: {"jmx_port":-1,"timestamp":"1413994171579","host":"ip-10-91-142-54.eu-west-1.compute.internal","version":1,"port":9092} (kafka.utils.ZkUtils$)
      [2014-10-22 16:37:22,736] INFO I wrote this conflicted ephemeral node [{"jmx_port":-1,"timestamp":"1413995842465","host":"ip-10-91-142-54.eu-west-1.compute.internal","version":1,"port":9092}] at /brokers/ids/0 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
      [2014-10-22 16:37:25,010] ERROR Error handling event ZkEvent[Data of /controller changed sent to kafka.server.ZookeeperLeaderElector$LeaderChangeListener@a6af882] (org.I0Itec.zkclient.ZkEventThread)
      java.lang.IllegalStateException: Kafka scheduler has not been started
              at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
              at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
              at kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
              at kafka.controller.KafkaController$$anonfun$2.apply$mcV$sp(KafkaController.scala:162)
              at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply$mcZ$sp(ZookeeperLeaderElector.scala:138)
              at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
              at kafka.server.ZookeeperLeaderElector$LeaderChangeListener$$anonfun$handleDataDeleted$1.apply(ZookeeperLeaderElector.scala:134)
              at kafka.utils.Utils$.inLock(Utils.scala:535)
              at kafka.server.ZookeeperLeaderElector$LeaderChangeListener.handleDataDeleted(ZookeeperLeaderElector.scala:134)
              at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:549)
              at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
      [2014-10-22 16:37:28,757] INFO Registered broker 0 at path /brokers/ids/0 with address ip-10-91-142-54.eu-west-1.compute.internal:9092. (kafka.utils.ZkUtils$)
      [2014-10-22 16:37:28,849] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
      [2014-10-22 16:38:56,718] INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor)
      [2014-10-22 16:38:56,850] INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor)
      [2014-10-22 16:38:56,985] INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor)
      

      The last log line repeats forever and is correlated with errors on the app side.
      Restarting Kafka fixes the errors.

      Steps to reproduce (with help from the mailing list):

      1. start zookeeper
      2. start kafka-broker
      3. create topic or start a producer writing to a topic
      4. stop zookeeper
      5. stop kafka-broker( kafka broker shutdown goes into WARN Session
        0x14938d9dc010001 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) java.net.ConnectException: Connection refused)
      6. kill -9 kafka-broker
      7. restart zookeeper and than kafka-broker leads into the the error above

        Attachments

        1. KAFKA-1724.patch
          1 kB
          Sriharsha Chintalapani

          Issue Links

            Activity

              People

              • Assignee:
                sriharsha Sriharsha Chintalapani
                Reporter:
                hakman Ciprian Hacman
                Reviewer:
                Jun Rao
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: