Major Description
We've seen a broker become unresponsive several times in Production.
Analysis of a thread dump showed a deadlock between two threads, namely:
- a thread that is calling SortedQueue.enqueue, which has already locked the SortedQueue and is trying to lock the Subscription's sendLock
- a thread that is closing a session by calling AMQChannel.unsubscribeAllConsumers, which has already locked the Subscription's sendLock and is trying to lock the SortedQueue.
The suggested fix is for SortedQueue.enqueue to not be synchronized, thereby removing the possibility of this deadlock occuring. SortedQueue.enqueue should instead lock on a private Object.
Note that SortedQueue is the only queue that is susceptible to this deadlock because it is the only one that has a synchronized enqueue method.