[AMQ-3421] Deadlock when queue fills up

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 5.5.0
• Fix Version/s: NEEDS_REVIEW

Environment

Tomcat 6.0.29, Spring 3.0.5, Oracle Java 6, Centos 5

• Assignee: Unassigned
• Reporter: Robert Elliot
• Created: 28/Jul/11 13:34
• Updated: 24/Jun/14 18:46
• Resolved: 24/Jun/14 18:46

Description

We are running a queue to do asynch audit updates, configured via Spring 3.0.5.

When the queue fills up Tomcat locks up with all catalina threads waiting on an object monitor in Spring. This object monitor is held by the "ActiveMQ Connection Executor: vm://localhost#2986" thread which is itself blocked for ever awaiting the stopped CountDownLatch at TransportConnection.stop(TransportConnection.java:930).

There are no "ActiveMQ Task" threads running, which suggests that either the task created by stopAsync has completed or did not run.

A code review leaves us baffled as to how this latch cannot have counted down, but it hasn't. Could Tomcat possibly be silently discarding the thread that was meant to do the stop without throwing an exception?! It seems unlikely but (as I understand it) TaskRunnerFactory is breaking the Servlet spec by running up its own Threads.