[AMQ-1824] Single producer and multiple consumer processes, some consumers stop getting messages.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Not A Problem
• Affects Version/s: 5.1.0
• Fix Version/s: 5.2.0

Environment

Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05

• Assignee: Gary Tully
• Reporter: Jerry Shea
• Created: 25/Jun/08 03:22
• Updated: 06/Nov/08 10:47
• Resolved: 03/Nov/08 12:04

Description

With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:

2008-06-25 12:07:29,297 [pool-1-thread-7] WARN  Transaction  - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
javax.transaction.xa.XAException: The connection is already closed
	at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
	at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
	at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
	at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
	at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
	at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
	at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
	at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
	at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
	at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
	at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
	at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
	at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
	at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
	at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
	at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
	at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
	at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
	at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
	at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
	at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
	... 19 more
2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction  - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
javax.transaction.xa.XAException: The connection is already closed
	at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
	at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
	at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
	at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
	at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
	at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
	at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
	at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
	at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
	at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
	at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
	at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
	at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
	at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
	at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
	at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
	at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
	at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
	... 16 more
2008-06-25 12:07:29,297 [pool-1-thread-7] INFO  org.apache.activemq.ra.ServerSessionImpl:0  - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure

I've written some code to reproduce this problem. Instructions:
1. unpack the attached source
2. build with 'mvn compile package'
3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.