[ARTEMIS-982] Deadlock on AMQP Close versus Delivery on AMQP

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.3
• Fix Version/s: 1.5.4, 2.0.0
• Assignee: Clebert Suconic
• Reporter: Clebert Suconic
• Created: 20/Feb/17 04:56
• Updated: 24/Feb/17 15:04
• Resolved: 24/Feb/17 15:04

Description

During the failover, there was this following thread:
"Thread-14 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@2697e5b4-3045305)" #153 prio=5 os_prio=0 tid=0x00007fb8c4002000 nid=0x5ce7 waiting on condition [0x00007fb8ac055000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000dcd99590> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
  at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
  at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.lockDelivery(ServerConsumerImpl.java:636)
  at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.setStarted(ServerConsumerImpl.java:613)
• locked <0x00000000dcdaebf0> (a java.lang.Object)
  at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.setStarted(ServerSessionImpl.java:1584)
  at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:345)
• locked <0x00000000dcd996f0> (a org.apache.activemq.artemis.core.server.impl.ServerSessionImpl)
  at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1237)
  at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:181)
  at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:130)
  at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:1229)
  at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.close(AMQPSessionCallback.java:317)
  at org.apache.activemq.artemis.protocol.amqp.proton.AMQPSessionContext.close(AMQPSessionContext.java:147)
  at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext$LocalListener.onRemoteClose(AMQPConnectionContext.java:339)
• locked <0x00000000dcdb4120> (a java.lang.Object)
  at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:43)
  at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:345)
  at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.access$000(ProtonHandler.java:43)
  at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler$1.run(ProtonHandler.java:62)
  at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

Threads like this will be waiting for the lock ^^^ :

It won't ever be satisfied.

"Thread-11 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@2697e5b4-3045305)" #150 prio=5 os_prio=0 tid=0x00007fb8c0010800 nid=0x5ce4 waiting for monitor entry [0x00007fb8ac359000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:607)

• waiting to lock <0x00000000dcdb4120> (a java.lang.Object)
  at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.sendMessage(AMQPSessionCallback.java:502)
  at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1022)
  at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:416)
  at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2635)
  at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2136)
  at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1700(QueueImpl.java:103)
  at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2868)
• locked <0x00000000c10ebda0> (a org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner)
  at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: GitHub Pull Request #1030