[AMQ-1564] Deadlock between VMTransport threads accessing AMQStore

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 5.1.0
• Fix Version/s: 5.1.0
• Component/s: Message Store

Environment

Linux

• Assignee: Unassigned
• Reporter: Rainer Klute
• Created: 23/Jan/08 11:46
• Updated: 23/Jan/08 21:13
• Resolved: 23/Jan/08 21:13

Description

A some time and quite a lot of ActiveMQ patches I dared to run my favorite Reliability JUnit test (see AMQ-1490) on the current ActiveMQ snapshot (revision 614484). Unfortunately already the first test case failed with a deadlock after processing so many messages.

Here are the stack traces of what ''jconsole'' considered the threads involved in the deadlock. They obviously have to do with the AMQStore:

<pre>
Name: VMTransport
State: BLOCKED on org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore@9f339e owned by: VMTransport
Total blocked: 341 Total waited: 3.750

Stack trace:
org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.acknowledgeReference(KahaTopicReferenceStore.java:127)
org.apache.activemq.store.amq.AMQTopicMessageStore.acknowledge(AMQTopicMessageStore.java:154)
org.apache.activemq.store.amq.AMQTopicMessageStore$1.afterCommit(AMQTopicMessageStore.java:113)

• locked org.apache.activemq.store.amq.AMQTopicMessageStore@edf389
  org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
  org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:68)
  org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
  org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
  org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
  org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
  org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:396)
  org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
  org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:280)
  org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
  org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:100)
  org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
  org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:202)
  org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
  org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  java.lang.Thread.run(Thread.java:619)
  </pre>

And the second one:

<pre>
Name: VMTransport
State: BLOCKED on org.apache.activemq.store.amq.AMQTopicMessageStore@edf389 owned by: VMTransport
Total blocked: 294 Total waited: 5.570

Stack trace:
org.apache.activemq.store.amq.AMQMessageStore.getLocation(AMQMessageStore.java:406)
org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:390)
org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:70)
org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:258)

• locked org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore@9f339e
  org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:64)
  org.apache.activemq.broker.region.cursors.TopicStorePrefetch.fillBatch(TopicStorePrefetch.java:215)
• locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
  org.apache.activemq.broker.region.cursors.TopicStorePrefetch.safeFillBatch(TopicStorePrefetch.java:200)
• locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
  org.apache.activemq.broker.region.cursors.TopicStorePrefetch.isEmpty(TopicStorePrefetch.java:96)
• locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
  org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.getNextCursor(StoreDurableSubscriberCursor.java:307)
• locked org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor@839267
  org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.hasNext(StoreDurableSubscriberCursor.java:190)
• locked org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor@839267
  org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:481)
• locked java.lang.Object@1ac539f
  org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:146)
  org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:178)
  org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
  org.apache.activemq.broker.region.Topic.dispatch(Topic.java:607)
  org.apache.activemq.broker.region.Topic$4.afterCommit(Topic.java:424)
  org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
  org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:68)
  org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
  org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
  org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
  org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
  org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:396)
  org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
  org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:280)
  org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
  org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:100)
  org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
  org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:202)
  org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
  org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  java.lang.Thread.run(Thread.java:619)
  </pre>