Qpid
  1. Qpid
  2. QPID-3446

Deadlock observed during broker shutdown with simultaneous closing of connection and attempted message delivery

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10, 0.11, 0.12
    • Fix Version/s: 0.14
    • Component/s: Java Broker, Java Client
    • Labels:
      None

      Description

      Deadlock is observed on simultaneous closing of connection on both client and broker when broker is shut down.

      Found one Java-level deadlock:
      =============================
      "Thread-0":
      waiting to lock monitor 0x0000000059d53cf8 (object 0x00000000bbea0000, a java.lang.Object),
      which is held by "MinaNetworkTransport(Acceptor)-12"
      "MinaNetworkTransport(Acceptor)-12":
      waiting for ownable synchronizer 0x00000000bbed8b10, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
      which is held by "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c"
      "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
      waiting to lock monitor 0x0000000059671c58 (object 0x00000000bbf11540, a [Lorg.apache.qpid.transport.Method,
      which is held by "MinaNetworkTransport(Acceptor)-12"

      Java stack information for the threads listed above:
      ===================================================
      "Thread-0":
      at org.apache.qpid.transport.Connection.close(Connection.java:586)

      • waiting to lock <0x00000000bbea0000> (a java.lang.Object)
        at org.apache.qpid.server.transport.ServerConnection.close(ServerConnection.java:269)
        at org.apache.qpid.server.connection.ConnectionRegistry.closeConnection(ConnectionRegistry.java:58)
        at org.apache.qpid.server.connection.ConnectionRegistry.close(ConnectionRegistry.java:50)
        at org.apache.qpid.server.virtualhost.VirtualHostImpl.close(VirtualHostImpl.java:574)
        at org.apache.qpid.server.virtualhost.VirtualHostRegistry.close(VirtualHostRegistry.java:105)
        at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:443)
        at org.apache.qpid.server.registry.ApplicationRegistry.close(ApplicationRegistry.java:470)
        at org.apache.qpid.server.registry.ConfigurationFileApplicationRegistry.close(ConfigurationFileApplicationRegistry.java:53)
        at org.apache.qpid.server.registry.ApplicationRegistry.remove(ApplicationRegistry.java:203)
        at org.apache.qpid.server.registry.ApplicationRegistry$ShutdownService.run(ApplicationRegistry.java:126)
        at java.lang.Thread.run(Thread.java:662)
        "MinaNetworkTransport(Acceptor)-12":
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000bbed8b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.apache.qpid.server.subscription.Subscription_0_10.getSendLock(Subscription_0_10.java:663)
        at org.apache.qpid.server.transport.ServerSession.unregister(ServerSession.java:409)
        at org.apache.qpid.server.transport.ServerSessionDelegate.closed(ServerSessionDelegate.java:1250)
        at org.apache.qpid.transport.Session.closed(Session.java:1041)
      • locked <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
        at org.apache.qpid.transport.Connection.closed(Connection.java:551)
      • locked <0x00000000bbea0000> (a java.lang.Object)
        at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:110)
        at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:202)
        at org.apache.qpid.server.protocol.ProtocolEngine_0_10.closed(ProtocolEngine_0_10.java:176)
        at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:86)
        at org.apache.qpid.transport.network.mina.MinaNetworkHandler.sessionClosed(MinaNetworkHandler.java:136)
        at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(AbstractIoFilterChain.java:550)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(AbstractIoFilterChain.java:269)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed(AbstractIoFilterChain.java:633)
        at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java:230)
        at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(ExecutorFilter.java:264)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
        at java.lang.Thread.run(Thread.java:662)
        "SubFlushRunner-org.apache.qpid.server.subscription.Subscription_0_10@6ce7ce4c":
        at org.apache.qpid.transport.Session.invoke(Session.java:586)
      • waiting to lock <0x00000000bbf11540> (a [Lorg.apache.qpid.transport.Method
        at org.apache.qpid.server.transport.ServerSession.sendMessage(ServerSession.java:203)
        at org.apache.qpid.server.subscription.Subscription_0_10.send(Subscription_0_10.java:602)
        at org.apache.qpid.server.queue.SimpleAMQQueue.deliverMessage(SimpleAMQQueue.java:747)
        at org.apache.qpid.server.queue.SimpleAMQQueue.attemptDelivery(SimpleAMQQueue.java:1666)
        at org.apache.qpid.server.queue.SimpleAMQQueue.flushSubscription(SimpleAMQQueue.java:1605)
        at org.apache.qpid.server.queue.SubFlushRunner.run(SubFlushRunner.java:58)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

      Found 1 deadlock.

        Activity

        Hide
        Alex Rudyy added a comment -

        Attached a patch to unregister existing subscriptions on connection close in order to avoid deadlocks.

        Show
        Alex Rudyy added a comment - Attached a patch to unregister existing subscriptions on connection close in order to avoid deadlocks.
        Hide
        Alex Rudyy added a comment -

        Robbie, could you please review and apply an attached patch with fixes?

        Show
        Alex Rudyy added a comment - Robbie, could you please review and apply an attached patch with fixes?
        Hide
        Robbie Gemmell added a comment -

        Patch applied.

        Show
        Robbie Gemmell added a comment - Patch applied.
        Hide
        Justin Ross added a comment -

        Reviewed and approved for 0.14.

        Show
        Justin Ross added a comment - Reviewed and approved for 0.14.
        Hide
        Robbie Gemmell added a comment -

        Merged to 0.14 branch.

        Show
        Robbie Gemmell added a comment - Merged to 0.14 branch.

          People

          • Assignee:
            Robbie Gemmell
            Reporter:
            Alex Rudyy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development