[QPID-3446] Deadlock observed during broker shutdown with simultaneous closing of connection and attempted message delivery

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10, 0.11, 0.12
• Fix Version/s: 0.14
• Component/s: Broker-J, JMS AMQP 0-x
• Assignee: Robbie Gemmell
• Reporter: Alex Rudyy
• Created: 22/Aug/11 12:38
• Updated: 11/Feb/15 20:06
• Resolved: 11/Nov/11 10:39

Description

Deadlock is observed on simultaneous closing of connection on both client and broker when broker is shut down.

Found one Java-level deadlock:
=============================
"Thread-0":
waiting to lock monitor 0x0000000059d53cf8 (object 0x00000000bbea0000, a java.lang.Object),
which is held by "MinaNetworkTransport(Acceptor)-12"
"MinaNetworkTransport(Acceptor)-12":
waiting for ownable synchronizer 0x00000000bbed8b10, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c"
"SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
waiting to lock monitor 0x0000000059671c58 (object 0x00000000bbf11540, a [Lorg.apache.qpid.transport.Method,
which is held by "MinaNetworkTransport(Acceptor)-12"

Java stack information for the threads listed above:
===================================================
"Thread-0":
at org.apache.qpid.transport.Connection.close(Connection.java:586)

• waiting to lock <0x00000000bbea0000> (a java.lang.Object)
  at org.apache.qpid.server.transport.ServerConnection.close(ServerConnection.java:269)
  at org.apache.qpid.server.connection.ConnectionRegistry.closeConnection(ConnectionRegistry.java:58)
  at org.apache.qpid.server.connection.ConnectionRegistry.close(ConnectionRegistry.java:50)
  at org.apache.qpid.server.virtualhost.VirtualHostImpl.close(VirtualHostImpl.java:574)
  at org.apache.qpid.server.virtualhost.VirtualHostRegistry.close(VirtualHostRegistry.java:105)
  at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:443)
  at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:470)
  at org.apache.qpid.server.registry.ConfigurationFileApplicationRegistry.close(ConfigurationFileApplicationRegistry.java:53)
  at org.apache.qpid.server.registry.ApplicationRegistry.remove(ApplicationRegistry.java:203)
  at org.apache.qpid.server.registry.ApplicationRegistry$ShutdownService.run(ApplicationRegistry.java:126)
  at java.lang.Thread.run(Thread.java:662)
  "MinaNetworkTransport(Acceptor)-12":
  at sun.misc.Unsafe.park(Native Method)
• parking to wait for <0x00000000bbed8b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
  at org.apache.qpid.server.subscription.Subscription_0_10.getSendLock(Subscription_0_10.java:663)
  at org.apache.qpid.server.transport.ServerSession.unregister(ServerSession.java:409)
  at org.apache.qpid.server.transport.ServerSessionDelegate.closed(ServerSessionDelegate.java:1250)
  at org.apache.qpid.transport.Session.closed(Session.java:1041)
• locked <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
  at org.apache.qpid.transport.Connection.closed(Connection.java:551)
• locked <0x00000000bbea0000> (a java.lang.Object)
  at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:110)
  at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:202)
  at org.apache.qpid.server.protocol.ProtocolEngine_0_10.closed(ProtocolEngine_0_10.java:176)
  at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:86)
  at org.apache.qpid.transport.network.mina.MinaNetworkHandler.sessionClosed(MinaNetworkHandler.java:136)
  at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(AbstractIoFilterChain.java:550)
  at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(AbstractIoFilterChain.java:269)
  at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilterChain.java:53)
  at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed(AbstractIoFilterChain.java:633)
  at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:230)
  at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
  at java.lang.Thread.run(Thread.java:662)
  "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
  at org.apache.qpid.transport.Session.invoke(Session.java:586)
• waiting to lock <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
  at org.apache.qpid.server.transport.ServerSession.sendMessage(ServerSession.java:203)
  at org.apache.qpid.server.subscription.Subscription_0_10.send(Subscription_0_10.java:602)
  at org.apache.qpid.server.queue.SimpleAMQQueue.deliverMessage(SimpleAMQQueue.java:747)
  at org.apache.qpid.server.queue.SimpleAMQQueue.attemptDelivery(SimpleAMQQueue.java:1666)
  at org.apache.qpid.server.queue.SimpleAMQQueue.flushSubscription(SimpleAMQQueue.java:1605)
  at org.apache.qpid.server.queue.SubFlushRunner.run(SubFlushRunner.java:58)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.