[QPID-5118] [JMS] Deadlock when closing a consumer and calling JMS operatons inside onMessage() on AMQP 0-8/0-9/0-9-1 connection

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22
• Fix Version/s: 0.31
• Component/s: JMS AMQP 0-x
• Assignee: Unassigned
• Reporter: Phil Harvey
• Created: 06/Sep/13 09:09
• Updated: 19/Mar/15 12:23
• Resolved: 10/Feb/15 09:49

Description

I have run the java systests with the JCarder agent attached. When using AMQP 0-9, JCarder detected a potential deadlock when the following are done concurrently:

• Close a consumer from an application thread.
• Call one of the JMS operations inside MessageListener.onMessage() that uses AMQConnectionDelegate_8_0.executeRetrySupport(..).

Here is a sketch of the deadlock scenario:

	Thread: Dispatcher-1-Conn-600
		AMQSession$Dispatcher.dispatchMessage()
			lock AMQSession.this._messageDeliveryLock
			...
				myMessageListener.onMessage()
					execute failover-aware operation eg createQueue, createProducer etc
						AMQConnectionDelegate_8_0.executeRetrySupport
							lock AMQConnection.getFailoverMutex()
	Thread: main
		BasicMessageConsumer.close()
			lock AMQSession.getFailoverMutex()
			lock AMQSession._messageDeliveryLock

Interestingly, AMQConnectionDelegate_0_10.executeRetrySupport(..) does not acquire the failover mutex (I have no idea if it should) so is not susceptible to this deadlock.

I've attached the GraphViz .dot file that JCarder generated when it detected this problem.

Note that this problem is similar to QPID-4574 so might be fixed by a single commit.

Issue Links

• is part of: QPID-6374 [Java Client] Eliminate deadlocks by acquiring locks in a consistent order
• is related to: QPID-4574 [JMS] Deadlock involving _failoverMutex and _messageDeliveryLock
• is related to: QPID-5119 [JMS] 3-thread deadlock on pre-AMQP-0-10 connection involving messageDeliveryLock, failover mutex and sessionCreationLock