ActiveMQ broker leaks advisory topics

• Key: AMQ-677
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Rob Davies
• Reporter: Andrew Lusk
• Component/s: Broker
• Fix Version/s: 4.0

Description

When I run the attached code, which AFAIK is completely legal JMS, the ActiveMQ broker grows to 500+ mb and crashes due to being out of heap space.

Some investigation with hprof has lead me to believe that the advisory topics created by the MessageConsumers (and Producers, but I use the same producer each time so that's not causing a problem) are being put into a DestinationMap and not being removed.

The rough origin of this is in the addProducer call in AdvisoryBroker, which creates the advisory topic.

Note that this memory is not freed when the DestinationInfo removing the original temptopic is received, nor when the actual client exits. The object lifetime of these advisory destinations seems very poorly defined. If they are implicitly created by the server, they should be implicitly destroyed by the same.

To reproduce, I've been running this code with -Dtopic=true and -Dmax=10000 (though the problem shows up well before this amount). This is just a modified version of the example ProducerTool (note it doesn't actually send any messages).

Please verify the correctness of the attached code.

Andrew Lusk

Comments

[Andrew Lusk]

I have reproduced this problem in svn trunk (previously I was a few weeks behind).

[Andrew Lusk]

Adding this line:

next.removeDestination(context, AdvisorySupport.getConsumerAdvisoryTopic(info.getDestination()), timeout);

below the fireAdvisory call in AdvisoryBroker::removeDestination appears to solve this problem, but at the cost of a very severe performance decrease.

Would really appreciate traction from someone more familiar with this particular code.

[Andrew Lusk]

Another solution that I've found to work, with no performance penalty, is to simple not create / send on advisory topics for temporary destinations. I'm not sure if this might break something else internally though.

[Rob Davies]

destinations that don't exist are added by calling the broker stack from the top to add the destination - which we don't need to do for advisories

[Rob Davies]

added fix suggested

[Andrew Lusk]

I got a fresh snapshot of trunk after your latest patch, and tested it with the code I attached above. It still exhibits this memory bloat problem, after several thousand iterations. Did this code work for you with this fix?

Also, the fix that was put in only removes consumer advisory topics, when producer advisory topics are just as much of a problem. The fix will also crash when info == null (when I mentioned putting that line in just after the fireAdvisory call, I meant still within the checked info != null block).

The code I provided was a stopgap solution that caused other serious problems for me (and maybe only worked because I was a few weeks back from trunk). In my tree I'm going with the second thing I mentioned (not creating advisory topics for temporary destinations) which is working fine for me while I work on investigating some other issues that have come up with higher numbers of iterations.

[Rob Davies]

I've checked in some updates and ran the program through JProfiler - I can't see any memory leaks.
Re - performance - usual practice is to create only a few (1 preferably) temp destination and use selectors - as the creation of any managed resource (consumer/producer/session/connection/destination is heavy compared to sending/selecting messages)

[Rob Davies]

correction - there's still a memory leak - gonna track it down

[Andrew Lusk]

The lines you added in r392929 throw an exception in the broker and crash my ProducerTool on the first iteration, since producer advisory topics and consumer advisory topics are unconditionally closed, when in the case of the above attached code it doesn't create any producers on the temporary topic, only consumers.

Please verify the fix on the above attached ProducerTool with -Dtopic=true and -Dmax=10000. It has always very readily reproduced this problem.

INFO Service - Sync error occurred: javax.jms.JMSException: Destination does not exist: topic://ActiveMQ.Advisory.Producer.Topic.ID:andrewlu-33101-1144699546209-0:0:1
javax.jms.JMSException: Destination does not exist: topic://ActiveMQ.Advisory.Producer.Topic.ID:andrewlu-33101-1144699546209-0:0:1
at org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:216)
at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:129)
at org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:180)
at org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:204)
at org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:214)
at org.apache.activemq.broker.AbstractConnection.processRemoveDestination(AbstractConnection.java:376)
at org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
at org.apache.activemq.broker.AbstractConnection.service(AbstractConnection.java:196)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:62)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:93)
at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:70)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:114)
at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:87)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:139)
at java.lang.Thread.run(Thread.java:595)

[Hiram Chirino]

Fixed in latest snapshot. I was also able to run your ProducerTool without issue.