Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-899

LeaderNotAvailableException the first time a new message for a partition is processed.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.2.0
    • Component/s: core
    • Labels:
      None

      Description

      I'm porting some unit tests from 0.7.2 to 0.8.0. The test does the following, all embedded in the same java process:

      – spins up a zk instance
      – spins up a kafka server using a fresh log directory
      – creates a producer and sends a message
      – creates a high-level consumer and verifies that it can consume the message
      – shuts down the consumer
      – stops the kafka server
      – stops zk

      The test seems to be working fine now, however, I consistently see the following exceptions (which from poking around the mailing list seem to be expected?). If these are expected, can we suppress the logging of these exceptions, since it clutters the output of tests, and presumably, clutters the logs of the running server/consumers, during clean startup and shutdown......

      When I call producer.send(), I get:

      1071 [main] WARN kafka.producer.BrokerPartitionInfo - Error while fetching metadata partition 0 leader: none replicas: isr: isUnderReplicated: false for topic partition [test-topic,0]: [class kafka.common.LeaderNotAvailableException]
      1081 [main] WARN kafka.producer.async.DefaultEventHandler - Failed to collate messages by topic,partition due to
      kafka.common.LeaderNotAvailableException: No leader for any partition
      at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartition(DefaultEventHandler.scala:212)
      at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:150)
      at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:148)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
      at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:148)
      at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:94)
      at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
      at kafka.producer.Producer.send(Producer.scala:74)
      at kafka.javaapi.producer.Producer.send(Producer.scala:32)
      at com.squareup.kafka.server.KafkaServerTest.produceAndConsumeMessage(KafkaServerTest.java:98)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
      at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
      at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
      at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
      1133 [kafka-request-handler-1] WARN kafka.server.HighwaterMarkCheckpoint - No highwatermark file is found. Returning 0 as the highwatermark for partition [test-topic,0]
      ...
      ...

      It would be great if instead of this exception, it would just log a meaningful message, like:

      "No leader was available for partition X, one will now be created"

      Jason

        Attachments

        1. kafka-899_v3.patch
          6 kB
          Jun Rao
        2. kafka-899_v2.patch
          5 kB
          Jun Rao
        3. kafka-899.patch
          5 kB
          Jun Rao

          Activity

            People

            • Assignee:
              junrao Jun Rao
              Reporter:
              jbrosenberg Jason Rosenberg
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: