Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3689

Exception when attempting to decrease connection count for address with no connections

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.1
    • Fix Version/s: None
    • Component/s: network
    • Labels:
      None
    • Environment:
    • Flags:
      Patch, Important

      Description

      As per Ismael Juma's suggestion in email thread to users@kafka.apache.org with the same subject, I am creating this bug report.

      The following error occurs in one of the brokers in our 3 broker cluster, which serves about 8000 topics. These topics are single partitioned with a replication factor = 3. Each topic gets data at a low rate – 200 bytes per sec. Leaders are balanced across the topics.

      Producers run from external servers (4 Ubuntu servers with same config as the brokers), each producing to 2000 topics utilizing kafka-python library.

      This error message occurs repeatedly in one of the servers. Between the hours of 10:30am and 1:30pm on 5/9/16, there were about 10 Million such occurrences. This was right after a cluster restart.

      This is not the first time we got this error in this broker. In those instances, error occurred hours / days after cluster restart.

      =====================================================

      [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
      java.lang.IllegalArgumentException: Attempted to decrease connection count for address with no connections, address: /X.Y.Z.144 (actual network address masked)
      at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
      at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
      at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
      at scala.collection.AbstractMap.getOrElse(Map.scala:59)
      at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
      at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
      at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
      at scala.collection.Iterator$class.foreach(Iterator.scala:742)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at kafka.network.Processor.run(SocketServer.scala:445)
      at java.lang.Thread.run(Thread.java:745)
      [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
      java.lang.IllegalArgumentException: Attempted to decrease connection count for address with no connections, address: /X.Y.Z.144
      at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
      at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
      at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
      at scala.collection.AbstractMap.getOrElse(Map.scala:59)
      at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
      at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
      at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
      at scala.collection.Iterator$class.foreach(Iterator.scala:742)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at kafka.network.Processor.run(SocketServer.scala:445)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

        1. kafka-3689-instrumentation.patch
          2 kB
          Jun Rao
        2. KAFKA-3689.log.redacted
          2.96 MB
          James Cheng

          Issue Links

            Activity

              People

              • Assignee:
                Ryan P Ryan P
                Reporter:
                buvana.ramanan@nokia.com Buvaneswari Ramanan
                Reviewer:
                Ismael Juma
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 72h
                  72h
                  Remaining:
                  Remaining Estimate - 72h
                  72h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified