Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
5.3.0
-
None
-
Java 1.6.0_07-b06 on Solaris (5.10) sparcv9
Description
I was performing an endurance test on ActiveMQ 5.3.0. My test consisted of a single queue with a number of producers and consumers. The producers send very simple text messages and the consumers simply receive the message and do no other processing. Using the OOTB configuration (which uses -Xmx512m) with JDBC persistence to an Oracle database, I found the system came to a halt at around 10.3 million messages. Looking at the VM revealed a very full heap and tiny gains from garbage collection. Restarting the broker allows the system to run again.
To determine the cause of the exhausted heap, I took a series of heap dumps over time. My examination of the heaps showed that the number of live instances of TreeMap$Entry and Long were increasing linearly with the number of messages. The Longs were owned by the TreeMap$Entry objects. The TreeMap$Entry objects could be tracked back to the TreeSet<Long> instance from the lastRecoveredMessageIds field in DefaultJDBCAdapter.
The only method that uses lastRecoveredMessageIds is:
public void doRecoverNextMessages(TransactionContext c, ActiveMQDestination destination, long nextSeq,
int maxReturned, JDBCMessageRecoveryListener listener) throws Exception
As the listener is called to recover a message, the id is added to this set. The id is only removed from this set if it is encountered on future run of doRecoverNextMessages when it is added to the cleanupIds list. The SQL that is executed at the beginning of the method filters messages based on having an id greater than nextSeq. If nextSeq is always large enough, an id is never added to cleanupIds and consequently never removed from lastRecoveredMessageIds.
I saw that the use of lastRecoveredMessageIds was introduced with AMQ-1918. Also, AMQ-2436 synchronizes the TreeSet, but that should have no effect on this issue.
Dejan mentioned some work done on JDBC persistence and a memory leak fix in association with AMQ-2519. I have performed the same test on the latest snapshots of 5.3.1 and 5.4. The same results were observed on those as well. Also, work done for AMQ-2519 modified JDBCMessageStore, JDBCTopicMessageStore, and JDBCPersistenceAdapter. I believe the issue is in DefaultJDBCAdapter.
To reproduce:
- Start a broker using JDBC persistence (sample configuration file will be attached)
- Start a number of producers and consumers using the same queue on that broker (sample WAR file will be attached. Modify WEB-INF/applicationContext.xml to set the queue name and broker URL. The war file currently uses TestQueue and tcp://localhost:61616, respectively.)
- Wait for about 20,000 messages to be processed (you can use less; it just gets easier in the heap dumps to see after about 20,000 messages. 29 bytes are retained per message that is processed)
- Look at the old generation size after a full garbage collection over time. It grows slowly.
- Obtain a heap dump. The heap dump will show a number of retained instances of TreeMap$Entry and Long. (sample heap dump will be attached)