1. Camel
  2. CAMEL-5683

JMS connection leak with request/reply producer on temporary queues


    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.10.0
    • Fix Version/s: 2.9.4, 2.10.2, 2.11.0
    • Component/s: camel-jms
    • Labels:
    • Environment:

      Apache Camel 2.10.0
      ActiveMQ 5.6.0
      Spring 3.2.1.RELEASE
      Java 1.6.0_27
      SunOS HOST 5.10 Generic_144488-09 sun4v sparc SUNW,SPARC-Enterprise-T5220

    • Estimated Complexity:


      Over time I see the number of temporary queues in ActiveMQ slowly climb. Using JMX information and memory dumps in MAT, I believe the cause is a connection leak in Apache Camel.

      My environment contains 2 ActiveMQ brokers in a network of brokers configuration. There are about 15 separate applications which use Apache Camel to connect to the broker using the ActiveMQ/JMS component. The various applications have different load profiles and route configurations.

      In the more active client applications, I found that ActiveMQ was listing 300+ consumers when, based on my configuration, I would expect no more than 75. The vast majority of the consumers are sitting on a temporary queue. Over time, the 300 number increments by one or two over about a 4 hour period.

      I did a memory dump on one of the more active client applications and found about 275 DefaultMessageListenerContainers. Using MAT, I can see that some of the containers are referenced by JmsProducers in the ProducerCache; however I can also see a large number of listener containers that are no longer being referenced at all. I was also able to match up a soft-references producer/listener endpoint with an unreferenced listener which means a second producer was created at some point.

      Looking through the ProducerCache code, it looks like the LRU cache uses soft-references to producers, in my case a JmsProducer. This seems problematic for two reasons:

      • If memory gets constrained and the GC cleans up a producer, it is never properly stopped.
      • If the cache gets full and the map removes the LRU producer, it is never properly stopped.

      What I believe is happening, is that my application is sending a few request/reply messages to a JmsProducer. The producer creates a TemporaryReplyManager which creates a DefaultMessageListenerContainer. At some point, the JmsProducer is claimed by the GC (either via the soft-reference or because the cache is full) and the reply manager is never stopped. This causes the listener container to continue to listen on the temporary queue, consuming local resources and more importantly, consuming resources on the JMS broker.

      I haven't had a chance to write an application to reproduce this behavior, but I will attach one of my route configurations and a screenshot of the MAT analysis looking at DefaultMessageListenerContainers. If needed, I could provide the entire memory dump for analysis (although I rather not post it publicly). The leak depends on memory usage or producer count in the client application because the ProducerCache must have some churn. Like I said, in our production system we see about 12 temporary queues abandoned per client per day.

      Unless I'm missing something, it looks like the producer cache would need to be much smarter to support stopping a producer when the soft-reference is reclaimed or a member of the cache is ejected from the LRU list.

      1. MAT Snapshot.png
        231 kB
        Michael Pilone
      2. Route Configuration.txt
        8 kB
        Michael Pilone
      3. Consumer List.txt
        95 kB
        Michael Pilone
        10 kB
        Michael Pilone
        10 kB
        Michael Pilone

        Issue Links


          Gavin made changes -
          Link This issue depends upon CAMEL-5688 [ CAMEL-5688 ]
          Gavin made changes -
          Link This issue depends on CAMEL-5688 [ CAMEL-5688 ]
          Claus Ibsen made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Claus Ibsen made changes -
          Link This issue depends on CAMEL-5688 [ CAMEL-5688 ]
          Michael Pilone made changes -
          Michael Pilone made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Claus Ibsen made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 2.9.4 [ 12323142 ]
          Fix Version/s 2.11.0 [ 12321695 ]
          Fix Version/s 2.10.2 [ 12323141 ]
          Resolution Fixed [ 1 ]
          Claus Ibsen made changes -
          Assignee Claus Ibsen [ davsclaus ]
          Michael Pilone made changes -
          Attachment [ 12547781 ]
          Michael Pilone made changes -
          Attachment Route Configuration.txt [ 12547566 ]
          Attachment Consumer List.txt [ 12547567 ]
          Michael Pilone made changes -
          Field Original Value New Value
          Attachment MAT Snapshot.png [ 12547565 ]
          Michael Pilone created issue -


            • Assignee:
              Claus Ibsen
              Michael Pilone
            • Votes:
              0 Vote for this issue
              3 Start watching this issue


              • Created: