[ARTEMIS-2881] Potential deadlock when destroying a queue and depaging concurrently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.15.0
• Assignee: Justin Bertram
• Reporter: Justin Bertram
• Created: 17/Aug/20 15:29
• Updated: 17/Aug/20 16:57
• Resolved: 17/Aug/20 16:57

Description

When a queue is destroyed (e.g. when a non-durable subscriber closes its connection) and depaging happen concurrently there can be a dead-lock which looks similar to ARTEMIS-2592.

Deadlock detected!

"Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501 BLOCKED on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1 owned by "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3078)
    -  blocked on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2600(QueueImpl.java:126)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl$DepageRunner.run(QueueImpl.java:4079)
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

    Number of locked synchronizers = 2
    - java.util.concurrent.ThreadPoolExecutor$Worker@5798e685
    - java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643


"Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643 owned by "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501
    at sun.misc.Unsafe.park(Native Method)
    -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:2073)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:2040)
    at org.apache.activemq.artemis.core.server.Queue.deleteMatchingReferences(Queue.java:296)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2030)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2025)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2228)
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2213)
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2161)
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2152)
    at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2132)
    at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
    at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$201/1675316695.run(Unknown Source)
    at org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
    at org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
    at org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1425)
    -  locked org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
    at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:633)
    at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:566)
    -  locked org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@115ee963
    at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:540)
    at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1707)
    at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:597)
    at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:292)
    at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$107/1704846988.onMessage(Unknown Source)
    at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
    at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6123280d

Issue Links

• links to: GitHub Pull Request #3243