Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6345

NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: clients
    • Labels:
      None

      Description

      example stack trace (code is ~0.10.2.*)

      java.util.ConcurrentModificationException: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
      	at java.util.HashMap$ValueIterator.next(HashMap.java:1458)
      	at org.apache.kafka.clients.InFlightRequests.inFlightRequestCount(InFlightRequests.java:109)
      	at org.apache.kafka.clients.NetworkClient.inFlightRequestCount(NetworkClient.java:382)
      	at org.apache.kafka.clients.producer.internals.Sender$SenderMetrics$1.measure(Sender.java:480)
      	at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:61)
      	at org.apache.kafka.common.metrics.KafkaMetric.value(KafkaMetric.java:52)
      	at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttribute(JmxReporter.java:183)
      	at org.apache.kafka.common.metrics.JmxReporter$KafkaMbean.getAttributes(JmxReporter.java:193)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:709)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705)
      

      looking at latest trunk, the code is still vulnerable:

      1. NetworkClient.inFlightRequestCount() eventually iterates over InFlightRequests.requests.values(), which is backed by a (non-thread-safe) HashMap
      2. this will be called from the "requests-in-flight" sensor's measure() method (Sender.java line ~765 in SenderMetrics ctr), which would be driven by some thread reading JMX values
      3. HashMap in question would also be updated by some client io thread calling NetworkClient.doSend() - which calls into InFlightRequests.add())

      i guess the only upside is that this exception will always happen on the thread reading the JMX values and never on the actual client io thread ...

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                smccauliff Sean McCauliff
                Reporter:
                radai radai rosenblatt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: