[QPID-5119] [JMS] 3-thread deadlock on pre-AMQP-0-10 connection involving messageDeliveryLock, failover mutex and sessionCreationLock

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22
• Fix Version/s: 0.31
• Component/s: JMS AMQP 0-x
• Assignee: Unassigned
• Reporter: Phil Harvey
• Created: 06/Sep/13 09:26
• Updated: 19/Mar/15 12:23
• Resolved: 10/Feb/15 09:50

Description

I have run the java systests with the JCarder agent attached. When using AMQP 0-9, JCarder detected a potential deadlock when the following are done concurrently:

• Close a consumer from an application thread.
• Call Connection.createSession from an application thread.
• Call Connection.createSession inside MessageListener.onMessage().

Here is a sketch of the deadlock scenario:

	"Dispatcher-1-Conn-780"
		AMQSession$Dispatcher.dispatchMessage()
			lock AMQSession.this._messageDeliveryLock
			AMQConnection.createSession()
				lock AMQConnection._sessionCreationLock BLOCKED BY applicationThread1
	"applicationThread1"
		AMQConnection.createSession
			lock AMQConnection._sessionCreationLock
			...
				AMQConnectionDelegate_8_0.executeRetrySupport
					lock AMQConnection.getFailoverMutex() BLOCKED BY applicationThread2
	"applicationThread2"
		BasicMessageConsumer.close()
			lock AMQSession.getFailoverMutex()
			lock AMQSession._messageDeliveryLock BLOCKED BY Dispatcher-1-Conn-780

Interestingly, AMQConnectionDelegate_0_10.executeRetrySupport(..) does not acquire the failover mutex (I have no idea if it should) so is not susceptible to this deadlock.

I've attached the GraphViz .dot file that JCarder generated when it detected this problem.

Note that this problem is similar to QPID-4574 and QPID-5118 so might be fixed by a single commit.

Issue Links

• is part of: QPID-6374 [Java Client] Eliminate deadlocks by acquiring locks in a consistent order
• relates to: QPID-4574 [JMS] Deadlock involving _failoverMutex and _messageDeliveryLock
• relates to: QPID-5118 [JMS] Deadlock when closing a consumer and calling JMS operatons inside onMessage() on AMQP 0-8/0-9/0-9-1 connection