Kafka
  1. Kafka
  2. KAFKA-816

Reduce noise in Kafka server logs due to NotLeaderForPartitionException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:

      Description

      NotLeaderForPartitionException is logged at the ERROR level with a full stack trace. But really this is just an informational message on the server when a client with stale metadata sends requests to the wrong leader for a partition. This floods the logs either if there are many clients or few clients sending many topics (migration tool or mirror maker). This should probably be logged at WARN and without the stack trace

      1. KAFKA-816.jpg
        62 kB
        BalajiSeshadri
      2. kafka-816.patch
        4 kB
        Neha Narkhede
      3. kafka-816-v2.patch
        4 kB
        Neha Narkhede

        Issue Links

          Activity

          Hide
          Neha Narkhede added a comment -

          Balaji,

          That fix was to reduce the noise in the kafka server logs. The error you see is on the consumer when the leader for some partitions is failing over to another replica. Can you please move the issue to a separate JIRA?

          Show
          Neha Narkhede added a comment - Balaji, That fix was to reduce the noise in the kafka server logs. The error you see is on the consumer when the leader for some partitions is failing over to another replica. Can you please move the issue to a separate JIRA?
          Hide
          BalajiSeshadri added a comment -

          Error occuring still in 0.8 branch,pls refer screen shot attached.

          Show
          BalajiSeshadri added a comment - Error occuring still in 0.8 branch,pls refer screen shot attached.
          Hide
          BalajiSeshadri added a comment - - edited

          I see fix merged in the show log but it is not working.

          SHA-1: 51421fcc0111031bb77f779a6f6c00520d526a34

          • KAFKA-816 Reduce noise in Kafka server logs due to NotLeaderForPartitionException; reviewed by Jun Rao
          Show
          BalajiSeshadri added a comment - - edited I see fix merged in the show log but it is not working. SHA-1: 51421fcc0111031bb77f779a6f6c00520d526a34 KAFKA-816 Reduce noise in Kafka server logs due to NotLeaderForPartitionException; reviewed by Jun Rao
          Hide
          BalajiSeshadri added a comment -

          Please let me know the Git URL.

          Show
          BalajiSeshadri added a comment - Please let me know the Git URL.
          Hide
          Neha Narkhede added a comment -

          All 0.8 development is happening on the 0.8 branch. Please can you retry on that branch ?

          Show
          Neha Narkhede added a comment - All 0.8 development is happening on the 0.8 branch. Please can you retry on that branch ?
          Hide
          BalajiSeshadri added a comment - - edited

          Using the below trunk and i still see error happening.Please let us know if this can be fixed.

          https://github.com/apache/kafka.git

          [2013-04-25 16:47:08,924] WARN [console-consumer-24019_MERD7-21964-1366930009136-8b7f9eb7-leader-finder-thread], Failed to add fetcher for [mytopic,0] to broker id:0,host:MERD7-21964.echostar.com,port:9092 (kafka.consumer.ConsumerFetcherManager$$anon$1)
          kafka.common.NotLeaderForPartitionException
          at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
          at java.lang.Class.newInstance0(Class.java:372)
          at java.lang.Class.newInstance(Class.java:325)
          at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:72)
          at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:163)
          at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:61)
          at kafka.server.AbstractFetcherThread.addPartition(AbstractFetcherThread.scala:167)
          at kafka.server.AbstractFetcherManager.addFetcher(AbstractFetcherManager.scala:48)
          at kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:79)
          at kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:75)
          at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
          at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
          at scala.collection.Iterator$class.foreach(Iterator.scala:772)
          at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
          at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
          at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
          at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
          at kafka.consumer.ConsumerFetcherManager$$anon$1.doWork(ConsumerFetcherManager.scala:75)
          at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)

          We are evaluating Kafka for our new messaging system and we had tough time running in windows.

          We somehow managed to run 0.8 using cygwin but when we run the console producer/consumer,we are not getting messages from consumer.

          Please help us to fix this issue,this might not be related but its keeping on throwing this error on consumer side.

          Show
          BalajiSeshadri added a comment - - edited Using the below trunk and i still see error happening.Please let us know if this can be fixed. https://github.com/apache/kafka.git [2013-04-25 16:47:08,924] WARN [console-consumer-24019_MERD7-21964-1366930009136-8b7f9eb7-leader-finder-thread] , Failed to add fetcher for [mytopic,0] to broker id:0,host:MERD7-21964.echostar.com,port:9092 (kafka.consumer.ConsumerFetcherManager$$anon$1) kafka.common.NotLeaderForPartitionException at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at java.lang.Class.newInstance0(Class.java:372) at java.lang.Class.newInstance(Class.java:325) at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:72) at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:163) at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:61) at kafka.server.AbstractFetcherThread.addPartition(AbstractFetcherThread.scala:167) at kafka.server.AbstractFetcherManager.addFetcher(AbstractFetcherManager.scala:48) at kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:79) at kafka.consumer.ConsumerFetcherManager$$anon$1$$anonfun$doWork$3.apply(ConsumerFetcherManager.scala:75) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95) at scala.collection.Iterator$class.foreach(Iterator.scala:772) at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45) at scala.collection.mutable.HashMap.foreach(HashMap.scala:95) at kafka.consumer.ConsumerFetcherManager$$anon$1.doWork(ConsumerFetcherManager.scala:75) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) We are evaluating Kafka for our new messaging system and we had tough time running in windows. We somehow managed to run 0.8 using cygwin but when we run the console producer/consumer,we are not getting messages from consumer. Please help us to fix this issue,this might not be related but its keeping on throwing this error on consumer side.
          Hide
          Neha Narkhede added a comment -

          checked in after making the suggested change

          Show
          Neha Narkhede added a comment - checked in after making the suggested change
          Hide
          Jun Rao added a comment -

          +1 on patch v2. To make the comment clearer, suggest that we change the comment "// NOTE: Failed produce requests is not incremented here" to "// NOTE: Failed produce requests is not incremented for UnknownTopicOrPartitionException and NotLeaderForPartitionException" (ditto for the comments in readMessageSets()) before checking in.

          Show
          Jun Rao added a comment - +1 on patch v2. To make the comment clearer, suggest that we change the comment "// NOTE: Failed produce requests is not incremented here" to "// NOTE: Failed produce requests is not incremented for UnknownTopicOrPartitionException and NotLeaderForPartitionException" (ditto for the comments in readMessageSets()) before checking in.
          Hide
          Neha Narkhede added a comment -

          Moved the comments as per your suggestion

          Show
          Neha Narkhede added a comment - Moved the comments as per your suggestion
          Hide
          Neha Narkhede added a comment -

          Sure. And the comment is not added to handleOffsetRequest since there is no failed offset request rate that we track. Will fix the grammar while checkin.

          Show
          Neha Narkhede added a comment - Sure. And the comment is not added to handleOffsetRequest since there is no failed offset request rate that we track. Will fix the grammar while checkin.
          Hide
          Jun Rao added a comment -

          Thanks for the patch. Instead of duplicating the comments for each type of exception, is it better to add a comment at the beginning of catch to explain how different exceptions are handled? The comment should probably be added to handleOffsetRequest() too. Also, there is a grammar mistake in "Failed produce requests is ".

          Show
          Jun Rao added a comment - Thanks for the patch. Instead of duplicating the comments for each type of exception, is it better to add a comment at the beginning of catch to explain how different exceptions are handled? The comment should probably be added to handleOffsetRequest() too. Also, there is a grammar mistake in "Failed produce requests is ".
          Hide
          Neha Narkhede added a comment -

          Special cased NotLeaderForPartitionException and UnknownTopicOrPartitionException to do the following -

          • Print a warn message, not a stack trace at ERROR level
          • Not increment the failed request metric since that is meant to track a broker's failure to serve requests for a partition for which it is the leader
          Show
          Neha Narkhede added a comment - Special cased NotLeaderForPartitionException and UnknownTopicOrPartitionException to do the following - Print a warn message, not a stack trace at ERROR level Not increment the failed request metric since that is meant to track a broker's failure to serve requests for a partition for which it is the leader

            People

            • Assignee:
              Neha Narkhede
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development