[KAFKA-2134] Producer blocked on metric publish

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Won't Fix
• Affects Version/s: 0.8.2.1
• Component/s: producer
• Environment: debian7, java8
• Assignee: Jun Rao
• Reporter: Vamsi Subhash Achanta
• Created: 20/Apr/15 13:05
• Updated: 03/Oct/16 23:04
• Resolved: 03/Oct/16 23:04

Description

Hi,

We have a REST api to publish to a topic. Yesterday, we started noticing that the producer is not able to produce messages at a good rate and the CLOSE_WAITs of our producer REST app are very high. All the producer REST requests are hence timing out.

When we took the thread dump and analysed it, we noticed that the threads are getting blocked on JmxReporter metricChange. Here is the attached stack trace.

"dw-70 - POST /queues/queue_1/messages" #70 prio=5 os_prio=0 tid=0x00007f043c8bd000 nid=0x54cf waiting for monitor entry [0x00007f04363c7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)

• waiting to lock <0x00000005c1823860> (a java.lang.Object)
  at org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:182)
• locked <0x00000007a5e526c8> (a org.apache.kafka.common.metrics.Metrics)
  at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:165)
• locked <0x00000007a5e526e8> (a org.apache.kafka.common.metrics.Sensor)

When I looked at the code of metricChange method, it uses a synchronised block on an object resource and it seems that it is held by another.