MessageProducer hangs when using many concurrent sessions

• Key: AMQ-2294
• Type: Bug
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Puneet Gupta
• Component/s: Broker, JMS client
• Affects Version/s: 5.2.0

Description

If you create a client (producer) that creates 25 non-transacted sessions and concurrently starts sending messages. The producer will hang after sending exactly 2560 1K messages.

This behavior is very easily reproducible.

Comments

[Puneet Gupta]

The client seems to hang at org.apache.activemq.transport.FutureResponse.getResult()

java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x03fc6578> (a java.util.concurrent.locks.Abstra
  ctQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
  .await(AbstractQueuedSynchronizer.java:1925)
  at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:
  317)
  at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse
  .java:40)
  at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorr
  elator.java:80)
  at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnect
  ion.java:1233)
  at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
• locked <0x0454b128> (a java.lang.Object)
  at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProdu
  cer.java:227)
  at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessa
  geProducerSupport.java:269)
  at perf.harness.ProducerObj.sendMsg(ProducerObj.java:317)
  at perf.harness.ProducerObj.run(ProducerObj.java:237)
  at java.lang.Thread.run(Thread.java:619)

You indicated that you're using the default configuration which limits the size of all queues and topics to 5mb and, by default in ActiveMQ, producer flow control is enabled. So either the destination to which you're sending has filled up or the has flow control kicked in and paused the producers. Chances are this is the producer flow control.

[Puneet Gupta]

I did not mention that I have 25 Message Consumers also listening on the same queue. So there is no reason for the producers to hang as eventually the producers and consumers should achieve equilibrium message rate. The problem persists even after killing the producers and consumers and restarting the consumers.

[Bruce Snyder]

The consumers and/or producers are not sharing a Session object are they?

[Puneet Gupta]

No, The producer, server and consumer are all running on separate machines.

[Puneet Gupta]

Consumer Thread Dump is attached as well

[Bruce Snyder]

This could still be the broker invoking the producer flow control because the consumers are either not consuming messages are are too slow. Are the consumers actually consuming messages? If so, are they actually acknowledging messages? Please attach the code for your consumers.

[Puneet Gupta]

I am using DUPS_OK_ACKNOWLEDGE acknowledgement mode and I explicitly call acknowledge() only if the Acknowledge Mode is CLIENT_ACKNOWLEDGE.

This works ok if there is only 1 consumer on the queue. If I create multiple consumers I start to see the hang with Acknowledge Mode AUTO_ACKNOWLEDGE and DUPS_OK_ACKNOWLEDGE.

Here is the code:

public void onMessage ( Message aMessage )
{
try
{
if(aMessage != null)
{
System.out.println("received message" + aMessage);
if (session.getTransacted())
{
try
{ session.commit(); }catch(JMSException jmse)

{ System.err.println("Error committing transaction....see trace for more details"); jmse.printStackTrace(); }

}
}
else

{ if (session.getAcknowledgeMode() == Session.CLIENT_ACKNOWLEDGE) aMessage.acknowledge(); }

}catch (Exception e)

{ e.printStackTrace(); }

}

[Puneet Gupta]

Confirmed that this is only a problem with DUPS_OK_ACKNOWLEDGE and multiple sessions