[QPID-6374] [Java Client] Eliminate deadlocks by acquiring locks in a consistent order

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.31
• Component/s: JMS AMQP 0-x
• Assignee: Robert Godfrey
• Reporter: Robert Godfrey
• Created: 09/Feb/15 21:30
• Updated: 02/Oct/19 19:20
• Resolved: 11/Feb/15 10:59

Description

A number of JIRAs exist relating to deadlocks in the Java client.

Deadlocks can generally be avoided if locks are always acquired in the same order however the Java client threading model can make this hard.

Two locks which in combination cause particular issues are the messageDeliveryLock and the failoverMutex.

The semantic purpose of the messageDeliveryLock is to meet the requirements of JMS that session.close(), connection.close() and consumer.close() block while the application is receiving a message in onMessage() or from receive(). The lock should only be acquired from an application thread or from the onMessage dispatcher thread - it should not be held by a thread acting on behalf of processing caused by io processing (e.g. the broker closing the session or connection). The lock is session specific.

The semantic purpose of the failover mutex is to ensure that application actions which cause state change at the broker can are not engaged in while failover is occurring. The lock is connection wide. In general the lock should only be held by application threads, except for when failover is occurring, in which case the thread managing failover will hold the lock.

Since messageDeliveryLock has a narrower scope (session) than the failoverMutex (which is connection scoped) it makes sense to ensure that the messageDeliveryLock is always acquired first. Changing this ordering where necessary, and removing and path where the messageDeliveryLock can be acquired by a non-application thread will greatly reduce the number of deadlocks seen.

Other deadlocks can be eliminated by removing the failoverMutex being acquired by threads acting on behalf of io processing, and by only holding the sessionCreationLock when setting the closed state or when actually creating a session.

Issue Links

• incorporates: QPID-4922 If Consumer close() method is invoked while inside onMessage(), it should be excuted after onMessage() has completed.
• incorporates: QPID-4276 Java client - deadlock on concurrent close of consumer and connection
• incorporates: QPID-4574 [JMS] Deadlock involving _failoverMutex and _messageDeliveryLock
• incorporates: QPID-4906 If Session close() or closed() method is invoked while inside onMessage(), they should be excuted after onMessage() has completed.
• incorporates: QPID-5117 Java client deadlocks if connection is closed while onMessage() is creating a session
• incorporates: QPID-5118 [JMS] Deadlock when closing a consumer and calling JMS operatons inside onMessage() on AMQP 0-8/0-9/0-9-1 connection
• incorporates: QPID-5119 [JMS] 3-thread deadlock on pre-AMQP-0-10 connection involving messageDeliveryLock, failover mutex and sessionCreationLock
• is related to: QPID-6461 Closing a connection in a JMS ExceptionListener logs an InterruptedException