Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.3.0
    • Fix Version/s: 5.3.1, 5.4.0
    • Component/s: Message Store
    • Labels:
      None
    • Environment:

      Java 1.6.0_07-b06 on Solaris (5.10) sparcv9

      Description

      I was performing an endurance test on ActiveMQ 5.3.0. My test consisted of a single queue with a number of producers and consumers. The producers send very simple text messages and the consumers simply receive the message and do no other processing. Using the OOTB configuration (which uses -Xmx512m) with JDBC persistence to an Oracle database, I found the system came to a halt at around 10.3 million messages. Looking at the VM revealed a very full heap and tiny gains from garbage collection. Restarting the broker allows the system to run again.

      To determine the cause of the exhausted heap, I took a series of heap dumps over time. My examination of the heaps showed that the number of live instances of TreeMap$Entry and Long were increasing linearly with the number of messages. The Longs were owned by the TreeMap$Entry objects. The TreeMap$Entry objects could be tracked back to the TreeSet<Long> instance from the lastRecoveredMessageIds field in DefaultJDBCAdapter.

      The only method that uses lastRecoveredMessageIds is:
      public void doRecoverNextMessages(TransactionContext c, ActiveMQDestination destination, long nextSeq,
      int maxReturned, JDBCMessageRecoveryListener listener) throws Exception

      As the listener is called to recover a message, the id is added to this set. The id is only removed from this set if it is encountered on future run of doRecoverNextMessages when it is added to the cleanupIds list. The SQL that is executed at the beginning of the method filters messages based on having an id greater than nextSeq. If nextSeq is always large enough, an id is never added to cleanupIds and consequently never removed from lastRecoveredMessageIds.

      I saw that the use of lastRecoveredMessageIds was introduced with AMQ-1918. Also, AMQ-2436 synchronizes the TreeSet, but that should have no effect on this issue.

      Dejan mentioned some work done on JDBC persistence and a memory leak fix in association with AMQ-2519. I have performed the same test on the latest snapshots of 5.3.1 and 5.4. The same results were observed on those as well. Also, work done for AMQ-2519 modified JDBCMessageStore, JDBCTopicMessageStore, and JDBCPersistenceAdapter. I believe the issue is in DefaultJDBCAdapter.

      To reproduce:

      • Start a broker using JDBC persistence (sample configuration file will be attached)
      • Start a number of producers and consumers using the same queue on that broker (sample WAR file will be attached. Modify WEB-INF/applicationContext.xml to set the queue name and broker URL. The war file currently uses TestQueue and tcp://localhost:61616, respectively.)
      • Wait for about 20,000 messages to be processed (you can use less; it just gets easier in the heap dumps to see after about 20,000 messages. 29 bytes are retained per message that is processed)
      • Look at the old generation size after a full garbage collection over time. It grows slowly.
      • Obtain a heap dump. The heap dump will show a number of retained instances of TreeMap$Entry and Long. (sample heap dump will be attached)
      1. jdbc-test.xml
        3 kB
        Jim Harter
      2. AMQTest.war
        5.58 MB
        Jim Harter
      3. heap.zip
        4.21 MB
        Jim Harter

        Activity

        Jim Harter created issue -
        Hide
        Jim Harter added a comment -

        Attaching sample broker configuration file, test web application to produce and consume messages, and a zip of a heap dump showing the memory leak

        Show
        Jim Harter added a comment - Attaching sample broker configuration file, test web application to produce and consume messages, and a zip of a heap dump showing the memory leak
        Jim Harter made changes -
        Field Original Value New Value
        Attachment heap.zip [ 18795 ]
        Attachment AMQTest.war [ 18794 ]
        Attachment jdbc-test.xml [ 18793 ]
        Hide
        Dejan Bosanac added a comment -

        Scheduled it for 5.3.1

        Show
        Dejan Bosanac added a comment - Scheduled it for 5.3.1
        Dejan Bosanac made changes -
        Fix Version/s 5.4.0 [ 12110 ]
        Fix Version/s 5.3.1 [ 12183 ]
        Dejan Bosanac made changes -
        Assignee Dejan Bosanac [ dejanb ]
        Hide
        Dejan Bosanac added a comment -

        Resolved with svn revision 906054.

        The problematic lastRecoveredMessageIds is removed completely. It was just a temp workaround for duplicate messages. Now we have it implemented when messages are added to the store, so there's no need for this check.

        The commit also includes a fix for a cursor problem spotted during the test.

        Show
        Dejan Bosanac added a comment - Resolved with svn revision 906054. The problematic lastRecoveredMessageIds is removed completely. It was just a temp workaround for duplicate messages. Now we have it implemented when messages are added to the store, so there's no need for this check. The commit also includes a fix for a cursor problem spotted during the test.
        Dejan Bosanac made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]

          People

          • Assignee:
            Dejan Bosanac
            Reporter:
            Jim Harter
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development