Qpid
  1. Qpid
  2. QPID-3216

Deadlock between "_lock" (in AMQSession#DisptcherThread) and "_messageDeliveryLock" (in AMQSession.java)

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10
    • Fix Version/s: Future
    • Component/s: Java Client
    • Labels:

      Description

      As per the following thread dump you can clearly see the deadlock between "_lock" (in AMQSession#DisptcherThread) and "_messageDeliveryLock" (in AMQSession.java)
      This is another regression and I am working to identify the rev that caused it and possibly understand the circumstances surrounding the change.

      dispatcher thread already obtained "_lock" in dispatchMessage method.
      dispatcher thread waiting to take "_messageDeliveryLock" in the same method.

      main thread has already obtained "_messageDeliveryLock" in AMQSession.close method.
      main thread waiting to take "_lock" in DispatcherThread.rejectPending method.

        Found one Java-level deadlock:
        =============================
        "Dispatcher-Channel-0":
          waiting to lock monitor 0x0000002ac1e6f0b8 (object 0x0000002a9f040448, a java.lang.Object),
          which is held by "main"
        "main":
          waiting to lock monitor 0x0000002ac1e6f010 (object 0x0000002a9f03ba40, a java.lang.Object),
          which is held by "Dispatcher-Channel-0"
      
        Java stack information for the threads listed above:
        ===================================================
        "Dispatcher-Channel-0":
           at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3315)
           - waiting to lock<0x0000002a9f040448>  (a java.lang.Object)
           - locked<0x0000002a9f03ba40>  (a java.lang.Object)
           at org.apache.qpid.client.AMQSession$Dispatcher.access$900(AMQSession.java:3096)
           at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3089)
           at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:55)
           at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3243)
           at java.lang.Thread.run(Thread.java:619)
        "main":
           at org.apache.qpid.client.AMQSession$Dispatcher.rejectPending(AMQSession.java:3122)
           - waiting to lock<0x0000002a9f03ba40>  (a java.lang.Object)
           at org.apache.qpid.client.AMQSession.confirmConsumerCancelled(AMQSession.java:897)
           at org.apache.qpid.client.BasicMessageConsumer_0_10.sendCancel(BasicMessageConsumer_0_10.java:189)
           at org.apache.qpid.client.BasicMessageConsumer.close(BasicMessageConsumer.java:573)
           - locked<0x0000002a9ef30aa0>  (a java.lang.Object)
           at org.apache.qpid.client.AMQSession.closeConsumers(AMQSession.java:2464)
           at org.apache.qpid.client.AMQSession.closeProducersAndConsumers(AMQSession.java:2514)
           at org.apache.qpid.client.AMQSession.close(AMQSession.java:731)
           - locked<0x0000002a9f040448>  (a java.lang.Object)
           - locked<0x0000002a9ef30aa0>  (a java.lang.Object)
           at org.apache.qpid.client.AMQSession.close(AMQSession.java:708)
           at org.apache.qpid.client.AMQConnection.closeAllSessions(AMQConnection.java:1003)
           at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:897)
           - locked<0x0000002a9ef30aa0>  (a java.lang.Object)
           - locked<0x0000002a9ef30ab0>  (a java.lang.Object)
           at org.apache.qpid.client.AMQConnection.doClose(AMQConnection.java:886)
           - locked<0x0000002a9f040448>  (a java.lang.Object)
           - locked<0x0000002a9ef30ab0>  (a java.lang.Object)
           at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:870)
           at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:861)
           at org.apache.qpid.client.AMQConnection.close(AMQConnection.java:856)
           at org.apache.qpid.test.utils.QpidBrokerTestCase.tearDown(QpidBrokerTestCase.java:1118)
           at junit.framework.TestCase.runBare(TestCase.java:130)
           at org.apache.qpid.test.utils.QpidBrokerTestCase.runBare(QpidBrokerTestCase.java:234)
           at junit.framework.TestResult$1.protect(TestResult.java:106)
           at junit.framework.TestResult.runProtected(TestResult.java:124)
           at junit.framework.TestResult.run(TestResult.java:109)
           at junit.framework.TestCase.run(TestCase.java:118)
           at org.apache.qpid.test.utils.QpidTestCase.run(QpidTestCase.java:120)
           at junit.framework.TestSuite.runTest(TestSuite.java:208)
           at junit.framework.TestSuite.run(TestSuite.java:203)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:297)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:672)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:546)
      

        Issue Links

          Activity

          Hide
          Rajith Attapattu added a comment -

          I have a suspicion that my following commit may have triggered this deadlock.
          http://svn.apache.org/viewvc?view=revision&revision=1092510

          I am yet to confirm this, but wanted to share this in case anybody else is also looking at it.

          Show
          Rajith Attapattu added a comment - I have a suspicion that my following commit may have triggered this deadlock. http://svn.apache.org/viewvc?view=revision&revision=1092510 I am yet to confirm this, but wanted to share this in case anybody else is also looking at it.
          Hide
          Robbie Gemmell added a comment -

          Linking to JIRA with now-reverted change suspected to be the cause.

          Show
          Robbie Gemmell added a comment - Linking to JIRA with now-reverted change suspected to be the cause.
          Hide
          Robbie Gemmell added a comment -

          Updating with an unreleased version as fix-for.

          Show
          Robbie Gemmell added a comment - Updating with an unreleased version as fix-for.
          Hide
          rabii eugen added a comment -

          So... what is the solution for this problem? We are facing it right now.
          Update to a higher version of qpid-client libraries?

          Show
          rabii eugen added a comment - So... what is the solution for this problem? We are facing it right now. Update to a higher version of qpid-client libraries?

            People

            • Assignee:
              Rajith Attapattu
              Reporter:
              Rajith Attapattu
            • Votes:
              3 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development