Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-6122

Potential Deadlock when a duplicate message is read from the store for the DLQ destination

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.13.0
    • Fix Version/s: 5.13.1, 5.14.0
    • Component/s: Broker, Message Store
    • Labels:
      None

      Description

      Potential for deadlock when duplicates read in from store for DLQ destination itself and another producer is trying to send messages from the DLQ destination.

      It can result with threads in following state

      Found one Java-level deadlock:
      =============================
      "ActiveMQ Transport: tcp:///127.0.0.1:59885@59875":
        waiting for ownable synchronizer 0x00000007f6188490, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "ActiveMQ BrokerService[localhost] Task-1"
      "ActiveMQ BrokerService[localhost] Task-1":
        waiting for ownable synchronizer 0x00000007f6190cf8, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "ActiveMQ Transport: tcp:///127.0.0.1:59885@59875"
      
      Java stack information for the threads listed above:
      ===================================================
      "ActiveMQ Transport: tcp:///127.0.0.1:59885@59875":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000007f6188490> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
      	at org.apache.activemq.broker.region.Queue.toString(Queue.java:937)
      	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:830)
      	at org.apache.activemq.broker.region.Queue.send(Queue.java:728)
      	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:390)
      	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:455)
      	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
      	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
      	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
      	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)
      	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
      	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)
      	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
      	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
      	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
      	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
      	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
      	at java.lang.Thread.run(Thread.java:722)
      "ActiveMQ BrokerService[localhost] Task-1":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000007f6190cf8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:894)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1221)
      	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
      	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:828)
      	at org.apache.activemq.broker.region.Queue.send(Queue.java:728)
      	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:390)
      	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:455)
      	at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
      	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
      	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
      	at org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)
      	at org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)
      	at org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:762)
      	at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:318)
      	at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:318)
      	at org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:330)
      	at org.apache.activemq.broker.region.BaseDestination.duplicateFromStore(BaseDestination.java:801)
      	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.dealWithDuplicates(AbstractStoreCursor.java:134)
      	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:110)
      	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:385)
      	- locked <0x00000007f6188550> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
      	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:142)
      	- locked <0x00000007f6188550> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
      	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)
      	- locked <0x00000007f61885a8> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
      	at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1906)
      	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2127)
      	at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1608)
      	- locked <0x00000007f61885f8> (a java.lang.Object)
      	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
      	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      	at java.lang.Thread.run(Thread.java:722)
      
      Found 1 deadlock.
      

        Attachments

          Activity

            People

            • Assignee:
              tabish Timothy A. Bish
              Reporter:
              tabish Timothy A. Bish
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: