[ARTEMIS-2276] Deadlock between QueueImpl and TransactionImpl

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.4
• Fix Version/s: 2.7.0
• Component/s: Broker
• Assignee: Unassigned
• Reporter: Frederik Petersen
• Created: 13/Mar/19 15:36
• Updated: 10/May/19 07:42
• Resolved: 10/May/19 07:42

Description

Hi,

we just experienced another deadlock in artemis.

I am attaching the thread dump. The issue seems to be here:

 

"Thread-0 (activemq-netty-threads)" Id=43 BLOCKED on org.apache.activemq.artemis.core.transaction.impl.TransactionImpl@196f00e1 owned by "Thread-7 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@5bda80bf)" Id=87
 at org.apache.activemq.artemis.core.server.impl.QueueImpl.getRefsOperation(QueueImpl.java:1332)
 - blocked on org.apache.activemq.artemis.core.transaction.impl.TransactionImpl@196f00e1
 at org.apache.activemq.artemis.core.server.impl.QueueImpl.getRefsOperation(QueueImpl.java:1328)
 at org.apache.activemq.artemis.core.server.impl.QueueImpl.acknowledge(QueueImpl.java:1295)
 at org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.acknowledge(MessageReferenceImpl.java:207)
 at org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.acknowledge(MessageReferenceImpl.java:199)
 at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:874)
 - locked org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@31c578da
 at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:891)
 at org.apache.activemq.artemis.core.protocol.stomp.StompSession$1.run(StompSession.java:159)
 at org.apache.activemq.artemis.core.protocol.stomp.StompSession.afterDelivery(StompSession.java:111)
 at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:469)
 at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2936)
 at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliverDirect(QueueImpl.java:2921)
 - locked org.apache.activemq.artemis.core.server.impl.QueueImpl@6b7ed217
[...]
"Thread-7 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@5bda80bf)" Id=87 BLOCKED on org.apache.activemq.artemis.core.server.impl.QueueImpl@6b7ed217 owned by "Thread-0 (activemq-netty-threads)" Id=43
 at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterCommit(RefsOperation.java:165)
 - blocked on org.apache.activemq.artemis.core.server.impl.QueueImpl@6b7ed217
 at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:546)
 - locked org.apache.activemq.artemis.core.transaction.impl.TransactionImpl@196f00e1
[...]

 

Thread-0 locks QueueImpl@6b7ed217.
Thread-7 locks TransactionImpl@196f00e1.
Thread-0 waits for TransactionImpl@196f00e1.
Thread-7 waits for QueueImpl@6b7ed217.

We had a few mqtt producers, stomp consumers and stomp producers. A few hundred items/sec. deliverDirect disabled on mqtt as suggested by clebertsuconic here: https://github.com/apache/activemq-artemis/pull/2577#issuecomment-472070556

But I guess this dead lock is not the same as the one discussed there.

Thanks for your help.