[AMQ-6626] Deadlock caused by ActiveMQMessageConsumer.dispatch() due to onEvent invocation

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Component/s: JMS client
• Assignee: Unassigned
• Reporter: Facundo Velazquez
• Created: 14/Mar/17 13:11
• Updated: 21/Mar/17 16:13
• Resolved: 21/Mar/17 16:13

Description

Hi.
In Mule ESB,
I have detected a deadlock caused by the interaction between Spring and ActiveMQ.
Basically, there are two execution chains that are causing the deadlock:

First Chain: when a Message arrives to the JMS Queue:

1) ActiveMQMessageConsumer.dispatch() gets the lock of FifoMessageDispatchChannel.mutex.
2) ActiveMQMessageConsumer.dispatch() calls the onMessage() method without releasing the lock.
3) onMessage Mule Implementation calls org.springframework.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke() because It needs a toString of the used resource (in this case the connection) to logging purposes.
3) in ToString() method, org.springframework.SingleConnectionFactory.getConnection() method is called and It takes the org.springframework.SingleConnectionFactory.connectionMonitor lock to access to connection (that's a shared resource).

Lock ordering in this chain:

FifoMessageDispatchChannel.mutex lock -> org.springframework.SingleConnectionFactory.connectionMonitor lock

Second Chain: when A reconnection-strategy is applied:

1) Mule stops the current connection calling org.springframework.SingleConnectionFactory.localStop()
2) LocalStop takes the org.springframework.SingleConnectionFactory.connectionMonitor lock.
3) After a chain of calls: FifoMessageDispatchChannel.stop() is called.
4) FifoMessageDispatchChannel.stop() takes the FifoMessageDispatchChannel.mutex lock.

Lock Ordering in this case:

org.springframework.SingleConnectionFactory.connectionMonitor lock -> FifoMessageDispatchChannel.mutex lock

As you can see, The chains don't take care about lock ordering, and It's causing a deadlock.

I've sent a mail with same subject to the user list.
Although I agree the best and faster solution is to not use the string() method of the connection in log (because It's taking a lock), and I'll remove it in Mule code, I think would be desirable to not call onEvent method without realising the adquired lock, to avoid potential deadlocks.

Thanks.