Possible 4.0 consumer client memory leak?

• Key: AMQ-735
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Critical
• Assignee: James Strachan
• Reporter: Karthik Sethuraman
• Component/s: JMS client
• Affects Version/s: 4.0
• Fix Version/s: 4.0.1

Description

On running the attached test programs (one producer on a topic, one consumer on a topic), the consumer eventually encounters OutOfMemoryError after receiving around 64 MB of messages). Default JVM heap size is used (64 MB).
The problem does not happen with RC2 build.

The test programs allows specification of message payloads of different sizes and different message rates, as well number of producer/consumer threads/instances., topics and topic connection factories, via user environment variables.

http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4660556

Comments

[Feng Wang]

I have run JProbe and I suspect that maybe MessageDispatch has the memory leak. Please see the attached screen shot.

Feng

[James Strachan]

I'm not sure I understand those pictures. Is the number after the types the number of instances created? If so from those pictures it seems like you've got tons of JMS Session objects (124596 of them!). Why so many JMS sessions? Those are objects your code creates and manages; so I suspect its your code not closing them down and letting them garbage collect?

Whic one of the 2 programs is it you ran the profiler on?

[James Strachan]

BTW if those numbers are not the number of instances in the JVM - which could be true (they might be hashcodes rather than instance counts) - then the pictures are not helpful - we need to know how many instances of each type there are - i.e. to find a memory leak we need to know what objects are using up all the RAM

[Feng Wang]

I profiled DumpDumpJMSMsg.java because that is the one will throw "out of memory" error very quickly.

I have attached the image for the instance count (instance.png). As you can see, byte array took the most of memory. But the counts of MessageDispatch, ActiveMQObjectMessage and MessageId keep increasing. In the previouslyploaded pictures,
there was a loop in MessageDispatch reference graph. Usually it is a sign of memory leak.

I guess that in Release 4 you start to use concurrent packge from emory universiy (but not RC2 build). CopyOnWriteArrayList
is from concurrent package and take a look at middle.png. CopyOnWriteArrayList has a reference to object[1].

Since I am quite new to ActiveMQ and emory concurrent package (only 2 days), I am not sure where the problem is.

[James Strachan]

So having about 809 Message objects in RAM doesn't particularly seem like a memory leak. The ActiveMQ JMS client will cache a pre-fetch size of messages in RAM

http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html

the default for topics is about 4000 messages. You could set this to a really small value, like 10 if you like.

[James Strachan]

I'm not yet convinced there is any memory leak. If you can find some leak we can always reopen this issue. As I said on the mail list, to reduce RAM overhead try use optimizeAcknowledge=false and reduce the prefetch value.

[James Strachan]

Am gonna mark this one as fixed as I think the optimiseAcknowledge being disabled in 4.0.1 will further reduce your RAM footprint

[Niclas Hedhman]

Does that mean ActiveMQ will also not throw away acknowledged messages and hold them around as well?

We are looking at a 1 message per 2 seconds which are consumed in milliseconds, and yet (in 4.0.1) we are running out of memory. That is true for transacted/non-transacted as well as auto-acknowledge and client-acknowledge.
However, we also see that Lingo has no such problems, so perhaps we should try to look into the Lingo sources to check how it uses ActiveMQ. Any pointer?

Btw, not sure if spring-jms throws anything particular into the mix either...