ActiveMQ
  1. ActiveMQ
  2. AMQ-1564

Deadlock between VMTransport threads accessing AMQStore

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 5.1.0
    • Fix Version/s: 5.1.0
    • Component/s: Message Store
    • Labels:
      None
    • Environment:

      Linux

      Description

      A some time and quite a lot of ActiveMQ patches I dared to run my favorite Reliability JUnit test (see AMQ-1490) on the current ActiveMQ snapshot (revision 614484). Unfortunately already the first test case failed with a deadlock after processing so many messages.

      Here are the stack traces of what ''jconsole'' considered the threads involved in the deadlock. They obviously have to do with the AMQStore:

      <pre>
      Name: VMTransport
      State: BLOCKED on org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore@9f339e owned by: VMTransport
      Total blocked: 341 Total waited: 3.750

      Stack trace:
      org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.acknowledgeReference(KahaTopicReferenceStore.java:127)
      org.apache.activemq.store.amq.AMQTopicMessageStore.acknowledge(AMQTopicMessageStore.java:154)
      org.apache.activemq.store.amq.AMQTopicMessageStore$1.afterCommit(AMQTopicMessageStore.java:113)

      • locked org.apache.activemq.store.amq.AMQTopicMessageStore@edf389
        org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
        org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:68)
        org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
        org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
        org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
        org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
        org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:396)
        org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
        org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:280)
        org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
        org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:100)
        org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
        org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:202)
        org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
        org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        java.lang.Thread.run(Thread.java:619)
        </pre>

      And the second one:

      <pre>
      Name: VMTransport
      State: BLOCKED on org.apache.activemq.store.amq.AMQTopicMessageStore@edf389 owned by: VMTransport
      Total blocked: 294 Total waited: 5.570

      Stack trace:
      org.apache.activemq.store.amq.AMQMessageStore.getLocation(AMQMessageStore.java:406)
      org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:390)
      org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
      org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:70)
      org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:258)

      • locked org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore@9f339e
        org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:64)
        org.apache.activemq.broker.region.cursors.TopicStorePrefetch.fillBatch(TopicStorePrefetch.java:215)
      • locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
        org.apache.activemq.broker.region.cursors.TopicStorePrefetch.safeFillBatch(TopicStorePrefetch.java:200)
      • locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
        org.apache.activemq.broker.region.cursors.TopicStorePrefetch.isEmpty(TopicStorePrefetch.java:96)
      • locked org.apache.activemq.broker.region.cursors.TopicStorePrefetch@1f5f472
        org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.getNextCursor(StoreDurableSubscriberCursor.java:307)
      • locked org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor@839267
        org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.hasNext(StoreDurableSubscriberCursor.java:190)
      • locked org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor@839267
        org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:481)
      • locked java.lang.Object@1ac539f
        org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:146)
        org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:178)
        org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
        org.apache.activemq.broker.region.Topic.dispatch(Topic.java:607)
        org.apache.activemq.broker.region.Topic$4.afterCommit(Topic.java:424)
        org.apache.activemq.transaction.Transaction.fireAfterCommit(Transaction.java:86)
        org.apache.activemq.transaction.LocalTransaction.commit(LocalTransaction.java:68)
        org.apache.activemq.broker.TransactionBroker.commitTransaction(TransactionBroker.java:170)
        org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
        org.apache.activemq.broker.BrokerFilter.commitTransaction(BrokerFilter.java:93)
        org.apache.activemq.broker.MutableBrokerFilter.commitTransaction(MutableBrokerFilter.java:103)
        org.apache.activemq.broker.TransportConnection.processCommitTransactionOnePhase(TransportConnection.java:396)
        org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:100)
        org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:280)
        org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
        org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:100)
        org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
        org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:202)
        org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
        org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        java.lang.Thread.run(Thread.java:619)
        </pre>

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Rainer Klute
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development