[AMQ-5785] Deadlock between FilePendingMessageCursor usage change and incoming send operations.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.10.0
• Component/s: Broker

Environment

Physical Machine (192 GB RAM, 24 VCPU), RHEL 5.9, Java 1.7
ActiveMQ runs on 4 GB heap

• Assignee: Timothy A. Bish
• Reporter: Sree Panchajanyam D
• Created: 20/May/15 13:21
• Updated: 26/Oct/16 08:38
• Resolved: 08/Mar/16 21:53

Description

During the peak loads we are encountering a recurring deadlock issue in ActiveMQ broker. The threads that are deadlocked are
ActiveMQ NIO Worker - trying to add message to FilePendingCursor
Broker.Service Worker - that is trying to expire message from FilePendingCursor.

=============================
Found one Java-level deadlock:
=============================
"ActiveMQ NIO Worker 1003":
waiting to lock monitor 0x00002aeeb515a4f8 (object 0x00000007807da3e8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
which is held by "ActiveMQ BrokerService.worker.1"
"ActiveMQ BrokerService.worker.1":
waiting for ownable synchronizer 0x000000077ac84b40, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "ActiveMQ NIO Worker 1003"

Java stack information for the threads listed above:
===================================================
"ActiveMQ NIO Worker 1003":
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)

• waiting to lock <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
  at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:96)
• locked <0x00000007784e8c88> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
  at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1855)
  at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:939)
  at org.apache.activemq.broker.region.Queue.send(Queue.java:733)
  at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:424)
  at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445)
  at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
  at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
  at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
  at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
  at org.apache.activemq.broker.UserIDBroker.send(UserIDBroker.java:56)
  at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
  at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:496)
  at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
  at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
  at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
  at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
  at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
  at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
  at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
  at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
  at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
  at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
  at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:744)
  "ActiveMQ BrokerService.worker.1":
  at sun.misc.Unsafe.park(Native Method)
• parking to wait for <0x000000077ac84b40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
  at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
  at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1841)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
• locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
  at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
• locked <0x00000007807da3e8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
  at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:744)