ActiveMQ
  1. ActiveMQ
  2. AMQ-3451

Tomcat 6.0.32 complains that ActiveMQ 5.5 doesn't shutdown a thread

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.5.0
    • Fix Version/s: 5.7.0
    • Component/s: JMS client
    • Labels:
    • Environment:

      jdk 1.6.0_23 for Linux 64 bit, Ubuntu 11.04
      Tomcat 6.0.32
      Spring 3.0.5

      Description

      Every time when restarting web application in Tomcat Manager I get messages:

      SEVERE: The web application [/sms] appears to have started a thread named [ActiveMQ Task-3] but has failed to stop it. This is very likely to create a memory leak.

      With every restart PermGen space is increased and finally i got OutOfMemory error for PermGen space.

      I use Spring DefaultMessageListenerContainer, and it's shutdown method closes properly receivers threads. What is "ActiveMQ Task-3" thread and how to close it properly ?

      1. transport_thread_leak.png
        497 kB
        Sergey
      2. patch.txt
        2 kB
        Sergey
      3. inherited_access_cotrol_context_leak.png
        292 kB
        Sergey
      4. AMQ-3541-3.patch
        28 kB
        Claus Ibsen
      5. AMQ-3451-broker2.patch
        11 kB
        Claus Ibsen
      6. AMQ-3451-broker.patch
        7 kB
        Claus Ibsen
      7. activemq-core-5.7-SNAPSHOT.jar
        3.84 MB
        Claus Ibsen

        Issue Links

          Activity

          Hide
          nimmi added a comment -

          ok,thank you.Will mail the user mailing list.

          Show
          nimmi added a comment - ok,thank you.Will mail the user mailing list.
          Hide
          Claus Ibsen added a comment -

          I created AMQ-4033 for the issue with the embedded AMQ broker. The JMS client now shutdown all the threads, and can be unloaded by the classloader etc.

          About the AMQ 5.7 release, then this should be discussed at the AMQ mailing list such as either the dev or user.

          Show
          Claus Ibsen added a comment - I created AMQ-4033 for the issue with the embedded AMQ broker. The JMS client now shutdown all the threads, and can be unloaded by the classloader etc. About the AMQ 5.7 release, then this should be discussed at the AMQ mailing list such as either the dev or user.
          Hide
          nimmi added a comment -

          We tried the 6th September build and our issues with ActiveMQ task runner threads have been fixed.Thank you.
          When can we expect the official version of ActiveMQ-5.7 to be released ?
          Also could anybody please comment on AMQ-4030.

          Show
          nimmi added a comment - We tried the 6th September build and our issues with ActiveMQ task runner threads have been fixed.Thank you. When can we expect the official version of ActiveMQ-5.7 to be released ? Also could anybody please comment on AMQ-4030 .
          Hide
          Claus Ibsen added a comment -

          Okay got further on the embedded broker side. Got a classloader leak fixed due a spring introspection cache, needed to be explicit cleared.
          The tomcat manager console now says there is no leaks. And all activemq and camel classes is fully unloaded when stopping the app.

          One last thing would be a JMX thread not being shutdown on the broker side.

          Show
          Claus Ibsen added a comment - Okay got further on the embedded broker side. Got a classloader leak fixed due a spring introspection cache, needed to be explicit cleared. The tomcat manager console now says there is no leaks. And all activemq and camel classes is fully unloaded when stopping the app. One last thing would be a JMX thread not being shutdown on the broker side.
          Hide
          Claus Ibsen added a comment -

          Okay got the JMS client leak fixed and committed to trunk. No leaks reported by Tomcat, and the WebappClassLoader did not have any strong references to any classes anymore. So seems we can fully undeploy/redeploy without leaks now.

          Show
          Claus Ibsen added a comment - Okay got the JMS client leak fixed and committed to trunk. No leaks reported by Tomcat, and the WebappClassLoader did not have any strong references to any classes anymore. So seems we can fully undeploy/redeploy without leaks now.
          Hide
          Claus Ibsen added a comment -

          Okay AMQ-4026 is now committed, so we have a better API for shutting down thread pools. Will work on getting the patches in the codebase, so the JMS Client work well in Tomcat when re-deploying apps.

          A fully embedded ActiveMQ broker in Tomcat has a number of more pools and other parts in the works, eg such as JMX and whatnot that can cause a bit pain, eg to ensure all JMX stuff is properly unregistered and whatnot.

          First priority is the JMS client though.

          Show
          Claus Ibsen added a comment - Okay AMQ-4026 is now committed, so we have a better API for shutting down thread pools. Will work on getting the patches in the codebase, so the JMS Client work well in Tomcat when re-deploying apps. A fully embedded ActiveMQ broker in Tomcat has a number of more pools and other parts in the works, eg such as JMX and whatnot that can cause a bit pain, eg to ensure all JMX stuff is properly unregistered and whatnot. First priority is the JMS client though.
          Hide
          Claus Ibsen added a comment -

          First part of patch is committed to the code base. The connector/transport code need a bit more love, as they tend to either create the pools in ctr / start or stop lifecycle methods. The code could possible be more consistent among them.

          Show
          Claus Ibsen added a comment - First part of patch is committed to the code base. The connector/transport code need a bit more love, as they tend to either create the pools in ctr / start or stop lifecycle methods. The code could possible be more consistent among them.
          Hide
          Claus Ibsen added a comment -

          I discovered a leak in Camel, when JMX is enabled.
          https://issues.apache.org/jira/browse/CAMEL-5564

          Show
          Claus Ibsen added a comment - I discovered a leak in Camel, when JMX is enabled. https://issues.apache.org/jira/browse/CAMEL-5564
          Hide
          Claus Ibsen added a comment -

          Patch with transport connections fixes. I also attached the core JAR for people to give a try. The other SNAPSHOT jars are in maven snapshot repo, where you can download them. As no fixes has yet been committed to the codebase.

          Show
          Claus Ibsen added a comment - Patch with transport connections fixes. I also attached the core JAR for people to give a try. The other SNAPSHOT jars are in maven snapshot repo, where you can download them. As no fixes has yet been committed to the codebase.
          Hide
          Claus Ibsen added a comment -

          With an embedded AMQ broker I still get this

          > The following web applications were stopped (reloaded, undeployed), but their
          classes from previous runs are still loaded in memory, thus causing a memory
          leak (use a profiler to confirm):
          /camel-example-activemq-tomcat

          Show
          Claus Ibsen added a comment - With an embedded AMQ broker I still get this > The following web applications were stopped (reloaded, undeployed), but their classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm): /camel-example-activemq-tomcat
          Hide
          Claus Ibsen added a comment -

          Okay got as far with a JMS client in Tomcat now not reporting any leaks from its console, when I stop the app or redeploy it.
          >
          Message:
          No web applications appear to have triggered a memory leak on stop, reload or undeploy.

          Show
          Claus Ibsen added a comment - Okay got as far with a JMS client in Tomcat now not reporting any leaks from its console, when I stop the app or redeploy it. > Message: No web applications appear to have triggered a memory leak on stop, reload or undeploy.
          Hide
          nimmi added a comment -

          Thank you very much for working on this issue.I'll check the builds page.

          Show
          nimmi added a comment - Thank you very much for working on this issue.I'll check the builds page.
          Hide
          Claus Ibsen added a comment -

          Patch with more fixes. Only the transports to go now.

          Show
          Claus Ibsen added a comment - Patch with more fixes. Only the transports to go now.
          Hide
          Claus Ibsen added a comment -

          The ASF CI server will once in a while deploy a new SNAPSHOT you can download. Keep an eye on
          https://builds.apache.org/job/ActiveMQ-Trunk-Deploy/

          The attached patch only fixes on the broker side, there is still all the transports. I am continuing on the patch, taking one step at a time. Currently testing with 2 more fixes for discover agent and another place. Then next is the remainder 7 transports using it.

          Show
          Claus Ibsen added a comment - The ASF CI server will once in a while deploy a new SNAPSHOT you can download. Keep an eye on https://builds.apache.org/job/ActiveMQ-Trunk-Deploy/ The attached patch only fixes on the broker side, there is still all the transports. I am continuing on the patch, taking one step at a time. Currently testing with 2 more fixes for discover agent and another place. Then next is the remainder 7 transports using it.
          Hide
          nimmi added a comment -

          Hello Claus,

          Would it be possible to attach the updated ActiveMQ jar please.Also let me know which version of ActiveMQ are you using.
          I am trying the 5.7.snapshot since it has the new DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown(); method but still Tomcat7 fails to shutdown cleanly.

          Show
          nimmi added a comment - Hello Claus, Would it be possible to attach the updated ActiveMQ jar please.Also let me know which version of ActiveMQ are you using. I am trying the 5.7.snapshot since it has the new DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown(); method but still Tomcat7 fails to shutdown cleanly.
          Hide
          Claus Ibsen added a comment -

          I have attached a patch to fix the part that is used by the broker, to use the executor the broker itself provides.

          Anyone see a problem with that? The unit tests passes, and a brief spin of running a standalone broker seems fine.

          Show
          Claus Ibsen added a comment - I have attached a patch to fix the part that is used by the broker, to use the executor the broker itself provides. Anyone see a problem with that? The unit tests passes, and a brief spin of running a standalone broker seems fine.
          Hide
          Claus Ibsen added a comment -

          The org.apache.activemq.thread.DefaultThreadPools seems to be used in 2 areas

          • by the broker a bit
          • transports

          The code for the broker, can be refactored to use a broker specific executor, so when the broker shutdown, those threads will be shutdown as well. That should be doable

          The transports can be used by JMS clients as well, so I wonder as Gary says, can we use a dedicated thread pool (executor service) per transport?
          If that would be doable, then we have start|stop callbacks on the transports, where we can ensure the thread pool is setup / shutdown properly.

          Show
          Claus Ibsen added a comment - The org.apache.activemq.thread.DefaultThreadPools seems to be used in 2 areas by the broker a bit transports The code for the broker, can be refactored to use a broker specific executor, so when the broker shutdown, those threads will be shutdown as well. That should be doable The transports can be used by JMS clients as well, so I wonder as Gary says, can we use a dedicated thread pool (executor service) per transport? If that would be doable, then we have start|stop callbacks on the transports, where we can ensure the thread pool is setup / shutdown properly.
          Hide
          Claus Ibsen added a comment -

          If people only run 1 broker in the JVM, we could as a temporary fix, add some flag/option/whatever to tell the broker, to shutdown these thread pools when the BrokerService is shutdown.

          Its of course a bit tricker when its a JMS client, as that would be some stop logic in a ConnectionFactory or the likes, to get a callback when to shutdown.

          Show
          Claus Ibsen added a comment - If people only run 1 broker in the JVM, we could as a temporary fix, add some flag/option/whatever to tell the broker, to shutdown these thread pools when the BrokerService is shutdown. Its of course a bit tricker when its a JMS client, as that would be some stop logic in a ConnectionFactory or the likes, to get a callback when to shutdown.
          Hide
          nimmi added a comment -

          When can we expect a solution to this issue ? We are facing the same issue on Tomcat7 as well.

          Show
          nimmi added a comment - When can we expect a solution to this issue ? We are facing the same issue on Tomcat7 as well.
          Hide
          Claudio Corsi added a comment -

          @Sergey

          The issue is more fundamental than your patch. Those threads are created by the default task runner factory and this factory is created once and then never shutdown throwout the life time of the process.

          Adding a shutdown of the default task runner will remove this issue but will raise another issue when using the shared classloader of the application server with multiple containers.

          The default task runner should be associated with the broker server and not a self contained static class. This should resolve the case of its use within the broker side code but will not resolve this issue within the client side issue. That needs a little more work on getting this to work properly.

          I have tried to shutdown the default task runner factory fix but when I hot deploy the container again. It raises a rejected exception when trying to add a task to the runner since this executor has been shutdown already.

          I tried a fix to associates a default task runner factory to the broker and that resolved the issue I was initially seeing.

          I still have an issue with the client code that uses the default task runner factory. This still needs work on my end and will probably require a different fix that would provide a similar solution. I was thinking about using WeakReferences associated with a ReferenceQueue. Whenever the remove method is called the task runner factory can be shutdown. This might work. I just need to implement this.

          --Claudio

          Show
          Claudio Corsi added a comment - @Sergey The issue is more fundamental than your patch. Those threads are created by the default task runner factory and this factory is created once and then never shutdown throwout the life time of the process. Adding a shutdown of the default task runner will remove this issue but will raise another issue when using the shared classloader of the application server with multiple containers. The default task runner should be associated with the broker server and not a self contained static class. This should resolve the case of its use within the broker side code but will not resolve this issue within the client side issue. That needs a little more work on getting this to work properly. I have tried to shutdown the default task runner factory fix but when I hot deploy the container again. It raises a rejected exception when trying to add a task to the runner since this executor has been shutdown already. I tried a fix to associates a default task runner factory to the broker and that resolved the issue I was initially seeing. I still have an issue with the client code that uses the default task runner factory. This still needs work on my end and will probably require a different fix that would provide a similar solution. I was thinking about using WeakReferences associated with a ReferenceQueue. Whenever the remove method is called the task runner factory can be shutdown. This might work. I just need to implement this. --Claudio
          Hide
          Sergey added a comment - - edited

          @liny

          I have downloaded the application from https://docs.google.com/open?id=0B-ZFcczN9qxYZmh5dDdoMFVCUkE, and analyzed the heap dump.

          I don't see any memory leaks related to threads, the only memory leak with your web application is AMQ-3959

          Show
          Sergey added a comment - - edited @liny I have downloaded the application from https://docs.google.com/open?id=0B-ZFcczN9qxYZmh5dDdoMFVCUkE , and analyzed the heap dump. I don't see any memory leaks related to threads, the only memory leak with your web application is AMQ-3959
          Hide
          liny added a comment -

          @Sergey

          My situation is described at http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654028.html with a simple web app. Would you like to try to see if you can see the same problem like me? Eclipse Memory Analysis looks a good tool, will try it.

          Show
          liny added a comment - @Sergey My situation is described at http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654028.html with a simple web app. Would you like to try to see if you can see the same problem like me? Eclipse Memory Analysis looks a good tool, will try it.
          Hide
          Sergey added a comment - - edited

          @liny

          I you didn't put ActiveMQ lib in tomcat shared lib, then I don't understand why you have a memory leak, because when I package ActiveMQ libs to WEB-INF/lib, then I don't have any memory leaks except AMQ-3959.
          You need to deploy your web application, then stop it in Tomcat Manager, then use Eclipse Memory Analysis tool, take a heap dump, open histogram, group by clasloader, then find paths to GC root for WebAppClassLoader. When done, you will know for sure the cause of the memory leak.

          Show
          Sergey added a comment - - edited @liny I you didn't put ActiveMQ lib in tomcat shared lib, then I don't understand why you have a memory leak, because when I package ActiveMQ libs to WEB-INF/lib, then I don't have any memory leaks except AMQ-3959 . You need to deploy your web application, then stop it in Tomcat Manager, then use Eclipse Memory Analysis tool, take a heap dump, open histogram, group by clasloader, then find paths to GC root for WebAppClassLoader. When done, you will know for sure the cause of the memory leak.
          Hide
          liny added a comment -

          @Sergey

          No, I didn't put ActiveMQ Lib in tomcat share lib directory. So your solutions work, right? If yes, really appreciated. I'll try soon!

          Show
          liny added a comment - @Sergey No, I didn't put ActiveMQ Lib in tomcat share lib directory. So your solutions work, right? If yes, really appreciated. I'll try soon!
          Hide
          liny added a comment -

          Hi,

          Is the solution is one of bellows?
          1. Killed all sessions, wait more than 30 secs, then thread pool will kill himself.
          2. Use patch.txt

          Show
          liny added a comment - Hi, Is the solution is one of bellows? 1. Killed all sessions, wait more than 30 secs, then thread pool will kill himself. 2. Use patch.txt
          Hide
          Sergey added a comment -

          Attached a patch which cleans the thread's context classloader and inherited security context.

          Show
          Sergey added a comment - Attached a patch which cleans the thread's context classloader and inherited security context.
          Hide
          Sergey added a comment - - edited

          I have tried the patch below
          -------------------------
          Index: activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java
          ===================================================================
          — activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java (revision 1366667)
          +++ activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java (working copy)
          @@ -108,6 +108,8 @@
          Thread thread = new Thread(runnable, name + "-" + id.incrementAndGet());
          thread.setDaemon(daemon);
          thread.setPriority(priority);
          + thread.setContextClassLoader(null);
          return thread;
          }
          });
          -------------------------

          Still have memory leak, now because of inherited access control context, see the screenshot.

          Show
          Sergey added a comment - - edited I have tried the patch below ------------------------- Index: activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java =================================================================== — activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java (revision 1366667) +++ activemq-core/src/main/java/org/apache/activemq/thread/TaskRunnerFactory.java (working copy) @@ -108,6 +108,8 @@ Thread thread = new Thread(runnable, name + "-" + id.incrementAndGet()); thread.setDaemon(daemon); thread.setPriority(priority); + thread.setContextClassLoader(null); return thread; } }); ------------------------- Still have memory leak, now because of inherited access control context, see the screenshot.
          Hide
          Sergey added a comment - - edited

          @Gary

          Ok, I will try to set tccl to null and then will run the memory profiler again.

          Show
          Sergey added a comment - - edited @Gary Ok, I will try to set tccl to null and then will run the memory profiler again.
          Hide
          Sergey added a comment - - edited

          Found another memory leak, see AMQ-3959

          Show
          Sergey added a comment - - edited Found another memory leak, see AMQ-3959
          Hide
          Gary Tully added a comment -

          hmm, ok. so it sounds like a sensible fix is to clear and reset the tccl in the thread factory such that it is not inherited by the activemq task runner threads.
          I just wonder if that behavior will break any clients, am thinking cxf and jaxb or in servicemix etc. It may need to be configurable via a system property just incase.
          fancy taking a stab at a patch?

          Show
          Gary Tully added a comment - hmm, ok. so it sounds like a sensible fix is to clear and reset the tccl in the thread factory such that it is not inherited by the activemq task runner threads. I just wonder if that behavior will break any clients, am thinking cxf and jaxb or in servicemix etc. It may need to be configurable via a system property just incase. fancy taking a stab at a patch?
          Hide
          Sergey added a comment -

          @Gary

          Yes, I understand the nature of the reference. When ActiveMQ thread pool creates a new thread, then the thread inherits the current thread context classloder. When the client web application is creating a JMS connection, then the context classloder is Tomcat WebAppClassLoader. As long as the thread is alive, it references web application classloader. The thread can be re-used by another web application while still holding a reference to the undeployed web application which created the thread.

          Show
          Sergey added a comment - @Gary Yes, I understand the nature of the reference. When ActiveMQ thread pool creates a new thread, then the thread inherits the current thread context classloder. When the client web application is creating a JMS connection, then the context classloder is Tomcat WebAppClassLoader. As long as the thread is alive, it references web application classloader. The thread can be re-used by another web application while still holding a reference to the undeployed web application which created the thread.
          Hide
          Gary Tully added a comment -

          @Sergey this is great information

          So the crux seems to be the classloader reference, if the pool is to be shared then it must not refer to an app specific loader. Do you understand the nature of that reference?

          The alternative approach is to remove the use of the statically defined default task runner factory from client side code and use one per connection factory.

          Show
          Gary Tully added a comment - @Sergey this is great information So the crux seems to be the classloader reference, if the pool is to be shared then it must not refer to an app specific loader. Do you understand the nature of that reference? The alternative approach is to remove the use of the statically defined default task runner factory from client side code and use one per connection factory.
          Hide
          Sergey added a comment -

          The problem is in org.apache.activemq.thread.TaskRunnerFactory#executor. The thread pool's keepAliveTime is 30 seconds. If you have a single web application in Tomcat which uses JMS, and
          you re-start (or re-deploy) your web application in an interval less that 30 seconds, then the thread is not terminated and re-used, the thread references the undeployed web applicaiton classloader and JVM cannot collect undeployed application and free PermGen memory area.

          Show
          Sergey added a comment - The problem is in org.apache.activemq.thread.TaskRunnerFactory#executor. The thread pool's keepAliveTime is 30 seconds. If you have a single web application in Tomcat which uses JMS, and you re-start (or re-deploy) your web application in an interval less that 30 seconds, then the thread is not terminated and re-used, the thread references the undeployed web applicaiton classloader and JVM cannot collect undeployed application and free PermGen memory area.
          Hide
          Sergey added a comment -

          Whether you have a memory leak or not depends on how fast you re-load or re-deploy applications and how intense ActiveMQ is used by other web applications on the same tomcat. Sometimes "ActiveMQ Transport" or "ActiveMQ Task" thread is terminated by inactivity, and in that case you don't have a memory leak. Most of the time you have a memory leak when deploying ActiveMQ to Tomcat shared libs.

          Show
          Sergey added a comment - Whether you have a memory leak or not depends on how fast you re-load or re-deploy applications and how intense ActiveMQ is used by other web applications on the same tomcat. Sometimes "ActiveMQ Transport" or "ActiveMQ Task" thread is terminated by inactivity, and in that case you don't have a memory leak. Most of the time you have a memory leak when deploying ActiveMQ to Tomcat shared libs.
          Hide
          Sergey added a comment - - edited

          You can't call org.apache.activemq.thread.DefaultThreadPools#shutdown when ActiveMQ libraries are in Tomcat shared libs, because calling that method will shutdown the shared thread pool, and all other web application deployed to the same Tomcat will be unable to send any JMS messages. You can do that when ActiveMQ libraries are packaged into web application only.

          Show
          Sergey added a comment - - edited You can't call org.apache.activemq.thread.DefaultThreadPools#shutdown when ActiveMQ libraries are in Tomcat shared libs, because calling that method will shutdown the shared thread pool, and all other web application deployed to the same Tomcat will be unable to send any JMS messages. You can do that when ActiveMQ libraries are packaged into web application only.
          Hide
          Sergey added a comment -

          However, when I removed ActiveMQ libraries from tomcat shared libraries and packaged them into WAR, the problem has gone because each web application now has its own ActiveMQ thread pool.

          Show
          Sergey added a comment - However, when I removed ActiveMQ libraries from tomcat shared libraries and packaged them into WAR, the problem has gone because each web application now has its own ActiveMQ thread pool.
          Hide
          Sergey added a comment -

          Attached a screenshot with profiler which clearly shows a memory leak.

          Show
          Sergey added a comment - Attached a screenshot with profiler which clearly shows a memory leak.
          Hide
          Sergey added a comment -

          The problem is that when you put ActiveMQ libraries into Tomcat shared "lib" folder, then ActiveMQ thread pool becomes shared between all web applications. When a web application sends a message, then a transport thread is created by a web application, then the thread is returned to the pool. Then the web application is undeployed, the thread remains alive in the pool and can be re-used by another applicaiton. In results into a memory leak, because the thread is still alive after undeployment of web application and references the web application context classloader.

          Show
          Sergey added a comment - The problem is that when you put ActiveMQ libraries into Tomcat shared "lib" folder, then ActiveMQ thread pool becomes shared between all web applications. When a web application sends a message, then a transport thread is created by a web application, then the thread is returned to the pool. Then the web application is undeployed, the thread remains alive in the pool and can be re-used by another applicaiton. In results into a memory leak, because the thread is still alive after undeployment of web application and references the web application context classloader.
          Hide
          liny added a comment - - edited

          @Gary
          Thanks for reply. Directly call below method in my stop() still can't work.

          ((PooledConnectionFactory) springHelper.getBean("pooledJmsFactory")).stop();
          

          Please refer to http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654084.html for details.

          Show
          liny added a comment - - edited @Gary Thanks for reply. Directly call below method in my stop() still can't work. ((PooledConnectionFactory) springHelper.getBean("pooledJmsFactory")).stop(); Please refer to http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654084.html for details.
          Hide
          Gary Tully added a comment -

          @liny, thanks for that test case war. Peeking at the code, I think the problem is related to the code in com.foo.MainApplication#stop

          That closes the connection, (returns them to the pool), but does not stop the connection pool. It does however stop the executors, and this prevents the real close of the connections when the spring context calls the custom stop destroy method. This is evident from the ERROR o.a.a.transport.tcp.TcpTransport - Could not stop service: tcp://foo.com/15.87.14.93:61616. Reason: java.util.concurrent.RejectedExecutionException exception.

          So you can explicitly call close on the connection pool, like:

          conn.close();
          ((PooledConnectionFactory) springHelper.getBean("pooledJmsFactory")).stop();
          DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown();
          DefaultThreadPools.shutdown();

          or somehow arrange it such that the calls to shutdown() are done when the spring context is destroyed. So after the connection pool is successfully destroyed.

          Show
          Gary Tully added a comment - @liny, thanks for that test case war. Peeking at the code, I think the problem is related to the code in com.foo.MainApplication#stop That closes the connection, (returns them to the pool), but does not stop the connection pool. It does however stop the executors, and this prevents the real close of the connections when the spring context calls the custom stop destroy method. This is evident from the ERROR o.a.a.transport.tcp.TcpTransport - Could not stop service: tcp://foo.com/15.87.14.93:61616. Reason: java.util.concurrent.RejectedExecutionException exception. So you can explicitly call close on the connection pool, like: conn.close(); ((PooledConnectionFactory) springHelper.getBean( "pooledJmsFactory" )).stop(); DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown(); DefaultThreadPools.shutdown(); or somehow arrange it such that the calls to shutdown() are done when the spring context is destroyed. So after the connection pool is successfully destroyed.
          Hide
          liny added a comment -

          I crated a war to reproduce this issue. Please refer to http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654028.html. Hope this is helpful.

          Show
          liny added a comment - I crated a war to reproduce this issue. Please refer to http://activemq.2283324.n4.nabble.com/Old-issue-Memory-leak-after-stop-web-application-in-Tomcat-tp4653959p4654028.html . Hope this is helpful.
          Hide
          liny added a comment -

          I have same issue after stop my application by Tomcat.
          I don't know why ActiveMQ team doesn't fix it for a long time.

          Show
          liny added a comment - I have same issue after stop my application by Tomcat. I don't know why ActiveMQ team doesn't fix it for a long time.
          Hide
          Guilhem RAMBAL added a comment -

          Hello,

          Have you got a workaround to solve this issue ?

          Sincerely

          Show
          Guilhem RAMBAL added a comment - Hello, Have you got a workaround to solve this issue ? Sincerely
          Hide
          Nils-Helge Garli added a comment -

          Same problem, using Active MQ 5.5.1. Tried with "closeAsync=false" which didn't seem to make any difference.

          Show
          Nils-Helge Garli added a comment - Same problem, using Active MQ 5.5.1. Tried with "closeAsync=false" which didn't seem to make any difference.
          Hide
          Dejan Bosanac added a comment -

          Looked at some similar issue, and it seems to me it's just Tomcat doesn't like threads created for async closing of activemq connections. Try using URLs like

          failover:(tcp://localhost:61616?closeAsync=false)

          and see if it helps.

          Show
          Dejan Bosanac added a comment - Looked at some similar issue, and it seems to me it's just Tomcat doesn't like threads created for async closing of activemq connections. Try using URLs like failover:(tcp: //localhost:61616?closeAsync= false ) and see if it helps.
          Hide
          Andreas Hartmann added a comment - - edited

          For me the following seems to work:

          Interrupt the TCP transport socket handler daemon thread:


          Index: activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java
          ===================================================================
          — activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java (revision 1214868)
          +++ activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java (working copy)
          @@ -380,6 +380,8 @@
          if (serverSocket != null)

          { serverSocket.close(); }

          + this.socketHandlerThread.interrupt();
          + this.socketHandlerThread = null;
          }


          Shutdown the task runner factory in a servlet context listener:


          public class CleanupAmqThreads implements ServletContextListener {

          @Override
          public void contextInitialized(ServletContextEvent sce) {
          }

          @Override
          public void contextDestroyed(ServletContextEvent sce)

          { DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown(); }

          }

          Show
          Andreas Hartmann added a comment - - edited For me the following seems to work: Interrupt the TCP transport socket handler daemon thread: Index: activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java =================================================================== — activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java (revision 1214868) +++ activemq-core/src/main/java/org/apache/activemq/transport/tcp/TcpTransportServer.java (working copy) @@ -380,6 +380,8 @@ if (serverSocket != null) { serverSocket.close(); } + this.socketHandlerThread.interrupt(); + this.socketHandlerThread = null; } Shutdown the task runner factory in a servlet context listener: public class CleanupAmqThreads implements ServletContextListener { @Override public void contextInitialized(ServletContextEvent sce) { } @Override public void contextDestroyed(ServletContextEvent sce) { DefaultThreadPools.getDefaultTaskRunnerFactory().shutdown(); } }
          Hide
          MBakhti added a comment -

          I have the same issue in Activemq 5.5.
          I tried :

          broker.getTaskRunnerFactory().shutdown();
          broker.getPersistenceTaskRunnerFactory().shutdown();
          broker.stop();
          broker.waitUntilStopped();
          Scheduler.shutdown();

          but it doesn't change anything

          Show
          MBakhti added a comment - I have the same issue in Activemq 5.5. I tried : broker.getTaskRunnerFactory().shutdown(); broker.getPersistenceTaskRunnerFactory().shutdown(); broker.stop(); broker.waitUntilStopped(); Scheduler.shutdown(); but it doesn't change anything
          Hide
          Javier Molina added a comment -

          In my case the message appears regardless of the queue(s) being empty.

          Using activemq 5.5 with camel 2.8 and spring 3.0.5 deployed to tomcat 6.0.32

          Broker is set up locally using activemq camel component.

          Show
          Javier Molina added a comment - In my case the message appears regardless of the queue(s) being empty. Using activemq 5.5 with camel 2.8 and spring 3.0.5 deployed to tomcat 6.0.32 Broker is set up locally using activemq camel component.
          Hide
          ravi bhatt added a comment - - edited

          and BTW i have tested this on tomcat 5, 6 and 7 with activemq 5.5.0

          The situation happens only when there were messages on the queue and messages where being processed inside onMessage of a listener. If there are no messages on the queue, it shuts down properly if we call connection close etc in cotextDestroyed.

          Show
          ravi bhatt added a comment - - edited and BTW i have tested this on tomcat 5, 6 and 7 with activemq 5.5.0 The situation happens only when there were messages on the queue and messages where being processed inside onMessage of a listener. If there are no messages on the queue, it shuts down properly if we call connection close etc in cotextDestroyed.
          Hide
          ravi bhatt added a comment -

          I am having similar issue where tomcat does not shut down properly because of activemq threads.

          I tried above solution org.apache.activemq.thread.DefaultThreadPools#shutdown, it does not help. I called this inside my contextDestroyed in tomcat, and it makes no difference to the situation.

          Show
          ravi bhatt added a comment - I am having similar issue where tomcat does not shut down properly because of activemq threads. I tried above solution org.apache.activemq.thread.DefaultThreadPools#shutdown, it does not help. I called this inside my contextDestroyed in tomcat, and it makes no difference to the situation.
          Hide
          Gary Tully added a comment -

          does calling: org.apache.activemq.thread.DefaultThreadPools#shutdown work for you? I guess it would need to be part of a javax.servlet.ServletContextListener or called as part of the shutdown of your message listener container.
          I guess we would need to include a little helper class to make it easier to configure.

          Show
          Gary Tully added a comment - does calling: org.apache.activemq.thread.DefaultThreadPools#shutdown work for you? I guess it would need to be part of a javax.servlet.ServletContextListener or called as part of the shutdown of your message listener container. I guess we would need to include a little helper class to make it easier to configure.
          Hide
          John Miller added a comment -

          Linked a similar issue, not sure if this is a duplicate.

          Show
          John Miller added a comment - Linked a similar issue, not sure if this is a duplicate.
          Hide
          John Miller added a comment -

          Duplicate

          Show
          John Miller added a comment - Duplicate
          Hide
          John Miller added a comment -
          Show
          John Miller added a comment - Duplicate of https://issues.apache.org/jira/browse/AMQ-2852
          Hide
          John Miller added a comment -

          My connection factory is declared in web.xml

          <!-- Standalone ActiveMQ -->
          <Resource name="jms/connectionFactory" auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory"
          description="Repository JMS Connection Factory" factory="org.apache.activemq.jndi.JNDIReferenceFactory"
          brokerURL="failover:(tcp://localhost:61616)?timeout=3000&jms.redeliveryPolicy.maximumRedeliveries=2&jms.redeliveryPolicy.initialRedeliveryDelay=3000L&jms.redeliveryPolicy.useExponentialBackOff=false"
          brokerName="LocalActiveMQBroker"/>

          Show
          John Miller added a comment - My connection factory is declared in web.xml <!-- Standalone ActiveMQ --> <Resource name="jms/connectionFactory" auth="Container" type="org.apache.activemq.ActiveMQConnectionFactory" description="Repository JMS Connection Factory" factory="org.apache.activemq.jndi.JNDIReferenceFactory" brokerURL="failover:(tcp://localhost:61616)?timeout=3000&jms.redeliveryPolicy.maximumRedeliveries=2&jms.redeliveryPolicy.initialRedeliveryDelay=3000L&jms.redeliveryPolicy.useExponentialBackOff=false" brokerName="LocalActiveMQBroker"/>

            People

            • Assignee:
              Claus Ibsen
              Reporter:
              John Miller
            • Votes:
              11 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development