Qpid
  1. Qpid
  2. QPID-3446

Deadlock observed during broker shutdown with simultaneous closing of connection and attempted message delivery

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10, 0.11, 0.12
    • Fix Version/s: 0.14
    • Component/s: Java Broker, Java Client
    • Labels:
      None

      Description

      Deadlock is observed on simultaneous closing of connection on both client and broker when broker is shut down.

      Found one Java-level deadlock:
      =============================
      "Thread-0":
      waiting to lock monitor 0x0000000059d53cf8 (object 0x00000000bbea0000, a java.lang.Object),
      which is held by "MinaNetworkTransport(Acceptor)-12"
      "MinaNetworkTransport(Acceptor)-12":
      waiting for ownable synchronizer 0x00000000bbed8b10, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
      which is held by "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c"
      "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
      waiting to lock monitor 0x0000000059671c58 (object 0x00000000bbf11540, a [Lorg.apache.qpid.transport.Method,
      which is held by "MinaNetworkTransport(Acceptor)-12"

      Java stack information for the threads listed above:
      ===================================================
      "Thread-0":
      at org.apache.qpid.transport.Connection.close(Connection.java:586)

      • waiting to lock <0x00000000bbea0000> (a java.lang.Object)
        at org.apache.qpid.server.transport.ServerConnection.close(ServerConnection.java:269)
        at org.apache.qpid.server.connection.ConnectionRegistry.closeConnection(ConnectionRegistry.java:58)
        at org.apache.qpid.server.connection.ConnectionRegistry.close(ConnectionRegistry.java:50)
        at org.apache.qpid.server.virtualhost.VirtualHostImpl.close(VirtualHostImpl.java:574)
        at org.apache.qpid.server.virtualhost.VirtualHostRegistry.close(VirtualHostRegistry.java:105)
        at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:443)
        at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:470)
        at org.apache.qpid.server.registry.ConfigurationFileApplicationRegistry.close(ConfigurationFileApplicationRegistry.java:53)
        at org.apache.qpid.server.registry.ApplicationRegistry.remove(ApplicationRegistry.java:203)
        at org.apache.qpid.server.registry.ApplicationRegistry$ShutdownService.run(ApplicationRegistry.java:126)
        at java.lang.Thread.run(Thread.java:662)
        "MinaNetworkTransport(Acceptor)-12":
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000bbed8b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.apache.qpid.server.subscription.Subscription_0_10.getSendLock(Subscription_0_10.java:663)
        at org.apache.qpid.server.transport.ServerSession.unregister(ServerSession.java:409)
        at org.apache.qpid.server.transport.ServerSessionDelegate.closed(ServerSessionDelegate.java:1250)
        at org.apache.qpid.transport.Session.closed(Session.java:1041)
      • locked <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
        at org.apache.qpid.transport.Connection.closed(Connection.java:551)
      • locked <0x00000000bbea0000> (a java.lang.Object)
        at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:110)
        at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:202)
        at org.apache.qpid.server.protocol.ProtocolEngine_0_10.closed(ProtocolEngine_0_10.java:176)
        at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:86)
        at org.apache.qpid.transport.network.mina.MinaNetworkHandler.sessionClosed(MinaNetworkHandler.java:136)
        at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(AbstractIoFilterChain.java:550)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(AbstractIoFilterChain.java:269)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed(AbstractIoFilterChain.java:633)
        at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:230)
        at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
        at java.lang.Thread.run(Thread.java:662)
        "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
        at org.apache.qpid.transport.Session.invoke(Session.java:586)
      • waiting to lock <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
        at org.apache.qpid.server.transport.ServerSession.sendMessage(ServerSession.java:203)
        at org.apache.qpid.server.subscription.Subscription_0_10.send(Subscription_0_10.java:602)
        at org.apache.qpid.server.queue.SimpleAMQQueue.deliverMessage(SimpleAMQQueue.java:747)
        at org.apache.qpid.server.queue.SimpleAMQQueue.attemptDelivery(SimpleAMQQueue.java:1666)
        at org.apache.qpid.server.queue.SimpleAMQQueue.flushSubscription(SimpleAMQQueue.java:1605)
        at org.apache.qpid.server.queue.SubFlushRunner.run(SubFlushRunner.java:58)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

      Found 1 deadlock.

        Activity

        Rob Godfrey made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robbie Gemmell made changes -
        Fix Version/s 0.14 [ 12316855 ]
        Fix Version/s 0.15 [ 12319043 ]
        Robbie Gemmell made changes -
        Summary Deadlock is observed on simultaneous closing of connection on both Java client and broker when java broker is shut down Deadlock observed during broker shutdown with simultaneous closing of connection and attempted message delivery
        Robbie Gemmell made changes -
        Status Ready To Review [ 10006 ] Resolved [ 5 ]
        Fix Version/s 0.15 [ 12319043 ]
        Resolution Fixed [ 1 ]
        Alex Rudyy made changes -
        Assignee Alex Rudyy [ alex.rufous ] Robbie Gemmell [ gemmellr ]
        Alex Rudyy made changes -
        Status In Progress [ 3 ] Ready To Review [ 10006 ]
        Alex Rudyy made changes -
        Alex Rudyy made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Alex Rudyy made changes -
        Assignee Robbie Gemmell [ gemmellr ] Alex Rudyy [ alex.rufous ]
        Alex Rudyy made changes -
        Field Original Value New Value
        Attachment deadlock.txt [ 12491200 ]
        Alex Rudyy created issue -

          People

          • Assignee:
            Robbie Gemmell
            Reporter:
            Alex Rudyy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development