[AMQ-4319] ActiveMQ destination are tracked by securitycontext and never cleaned up, causes ActiveMQ to crash.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.6.0
• Fix Version/s: 5.9.0

Environment

RHEL 6

• Assignee: Timothy A. Bish
• Reporter: Matt Woodson
• Created: 14/Feb/13 17:13
• Updated: 25/Feb/13 18:24
• Resolved: 25/Feb/13 18:24

Description

We are using ActiveMQ + mcollective. With mcollective, we are using the "activemq connector (http://docs.puppetlabs.com/mcollective/reference/plugins/connector_activemq.html)"

We are running ActiveMQ in the EC2 Cloud, so we have enabled security for the connections being made.

We have 2 ActiveMQ servers that are configures as network of brokers, about around 200 nodes that we communicate with.

The problem we are seeing is that after about 1 week in operation, ActiveMQ will crash because it runs out of memory. We have been monitoring the heap, and you will see it rise during the week, and then the wrapper will eventually restart activemq

wrapper.log

ERROR  | wrapper  | 2013/02/01 09:38:33 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR  | wrapper  | 2013/02/01 09:38:33 | JVM did not exit on request, terminated
INFO   | wrapper  | 2013/02/01 09:38:33 | JVM exited on its own while waiting to kill the application.
STATUS | wrapper  | 2013/02/01 09:38:33 | JVM exited in response to signal SIGKILL (9).
STATUS | wrapper  | 2013/02/01 09:38:38 | Launching a JVM...}

In the activemq connector page, it says that there needs to be cleanup of queues. This is put into place.

I did a heap dump, and with external help I was told that "the authentication plugin is retaining destinations in a cached connection"

These connections need to be deleted and cleaned up as well.