[KAFKA-3172] Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.9.0.0
• Component/s: consumer
• Environment: linux
• Flags: Important
• Assignee: Neha Narkhede
• Reporter: Dany Benjamin
• Reviewer: Jun Rao
• Created: 29/Jan/16 18:53
• Updated: 02/Jan/17 23:29

Description

When running multiple consumers on same group (400 - for a 400 partitioned topic), the application for all threads blocks at consumer.poll() method. The timeout parameter sent in is 1.

Stack dump:
"pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x00007f6bb6d53800 nid=0xc349 waiting on condition [0x00007f63df8f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x0000000605812710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

"kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 tid=0x00007f6bb6d52000 nid=0xc348 runnable [0x00007f63df9f8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000006058283e8> (a sun.nio.ch.Util$2)
• locked <0x00000006058283d8> (a java.util.Collections$UnmodifiableSet)
• locked <0x0000000605828390> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  at org.apache.kafka.common.network.Selector.select(Selector.java:425)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
  at java.lang.Thread.run(Thread.java:745)