[QPID-5050] Clients calling Connection#stop() (or #close()) from within an ExceptionListener have potential for deadlock

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8, 0.10, 0.12, 0.14, 0.16, 0.18, 0.20, 0.22
• Fix Version/s: 0.25
• Component/s: JMS AMQP 0-x
• Assignee: Keith Wall
• Reporter: Keith Wall
• Created: 07/Aug/13 15:51
• Updated: 02/Oct/19 19:21
• Resolved: 16/Aug/13 12:47

Description

A deadlock possibility exists for client applications calling Connection#stop within an application owned ExceptionListener.

Unfortunately a common messaging framework (Spring), installs such a ExceptionListener.

In a recent support call, such a deadlock had occurred between the dispatcher thread (whose onMessage was in the process of creating a session) and a pooled thread bouncing a message back to the application. (The bounced messages is returned to the application via the exception listener).

The deadlock involves the Dispatcher._lock and AMQConnection._failoverMutex.

The deadlock was reproduced with a system test (attached to Jira) and deadlock captured with jstack -l <pid> (output below).

Dispatcher-1-Conn-4                  pool-8-thread-1
 + acquires c888 (Dispatcher#_lock)    + acquires ca68 (AMQConnection#_failoverMutex)
 + tries to acquire ca68               + tries to acquire c888

Found one Java-level deadlock:
=============================
"pool-8-thread-1":
  waiting to lock monitor 0x000000005b0d3560 (object 0x00000000f70bc888, a java.lang.Object),
  which is held by "Dispatcher-1-Conn-4"
"Dispatcher-1-Conn-4":
  waiting to lock monitor 0x000000005b187308 (object 0x00000000f70bca68, a java.lang.Object),
  which is held by "pool-8-thread-1"

Java stack information for the threads listed above:
===================================================
"pool-8-thread-1":
        at org.apache.qpid.client.AMQSession$Dispatcher.setConnectionStopped(AMQSession.java:3276)
        - waiting to lock <0x00000000f70bc888> (a java.lang.Object)
        at org.apache.qpid.client.AMQSession.stop(AMQSession.java:2382)
        at org.apache.qpid.client.AMQConnection.stop(AMQConnection.java:835)
        at org.apache.qpid.test.unit.client.connection.ExceptionListenerTest$4.onException(ExceptionListenerTest.java:206)
        at org.apache.qpid.client.AMQConnection.exceptionReceived(AMQConnection.java:1329)
        - locked <0x00000000f70bca68> (a java.lang.Object)
        at org.apache.qpid.client.AMQSession_0_8$4.run(AMQSession_0_8.java:600)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
"Dispatcher-1-Conn-4":
        at org.apache.qpid.client.AMQConnectionDelegate_8_0.executeRetrySupport(AMQConnectionDelegate_8_0.java:333)
        - waiting to lock <0x00000000f70bca68> (a java.lang.Object)
        at org.apache.qpid.client.AMQConnection.executeRetrySupport(AMQConnection.java:624)
        at org.apache.qpid.client.failover.FailoverRetrySupport.execute(FailoverRetrySupport.java:102)
        at org.apache.qpid.client.AMQSession.createProducerImpl(AMQSession.java:2600)
        at org.apache.qpid.client.AMQSession.createProducer(AMQSession.java:1176)
        at org.apache.qpid.client.AMQSession.createProducer(AMQSession.java:98)
        at org.apache.qpid.test.unit.client.connection.ExceptionListenerTest$5.onMessage(ExceptionListenerTest.java:229)
        at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:744)
        at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:718)
        at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3388)
        at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3327)
        - locked <0x00000000f71747e0> (a java.lang.Object)
        - locked <0x00000000f70bc888> (a java.lang.Object)
        at org.apache.qpid.client.AMQSession$Dispatcher.access$900(AMQSession.java:3114)
        at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3107)
        at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54)
        at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3250)
        at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

Issue Links

• is related to: QPID-2756 Java Client dead locks on session close using spring CachingConnectionFactory