Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-rc1
    • Fix Version/s: 2.0-rc2
    • Component/s: Core
    • Labels:
    • Environment:

      Description

      Dynamically loading a JAR that uses log4j2 results in a memory leak when the JAR is unloaded. This can be observed by deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
      1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
      2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
      3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

      This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

      This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          4d 21h 15m 1 Matt Sicker 23/Mar/14 20:00
          Resolved Resolved Reopened Reopened
          7d 19h 42m 1 Remko Popma 31/Mar/14 16:43
          Reopened Reopened Resolved Resolved
          34d 7h 37m 1 Matt Sicker 05/May/14 00:20
          Hide
          Ralph Goers added a comment -

          I am in front of my computer now so I can look at code. PropertiesUtil looks for a resource named "log4j2.component.properties". We need to extend that so it can find multiple instances and concatenate them. Then when you look up a property it can either come from that property file or a system property. So if the web module has a property file named log4j2.component.properties with the property in it we will know Log4j is running in a servlet container. This is much better than relying on a class that someone might decide to rename.

          Show
          Ralph Goers added a comment - I am in front of my computer now so I can look at code. PropertiesUtil looks for a resource named "log4j2.component.properties". We need to extend that so it can find multiple instances and concatenate them. Then when you look up a property it can either come from that property file or a system property. So if the web module has a property file named log4j2.component.properties with the property in it we will know Log4j is running in a servlet container. This is much better than relying on a class that someone might decide to rename.
          Hide
          Matt Sicker added a comment -

          Wait so like a system property? Or we could even use a service via the META-INF/services/ thing.

          Show
          Matt Sicker added a comment - Wait so like a system property? Or we could even use a service via the META-INF/services/ thing.
          Hide
          Ralph Goers added a comment -

          Rather than check for a class I would prefer that we look for a property that can be found by PropertyUtil. The web module should have the appropriate file defined in it so the property is always defined when the jar is present.

          Show
          Ralph Goers added a comment - Rather than check for a class I would prefer that we look for a property that can be found by PropertyUtil. The web module should have the appropriate file defined in it so the property is always defined when the jar is present.
          Matt Sicker made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Matt Sicker added a comment -

          Shutdown thread memory leak fixed in r1592429.

          Show
          Matt Sicker added a comment - Shutdown thread memory leak fixed in r1592429.
          Hide
          Matt Sicker added a comment -

          In r1592429, I've added a check for the presence of a class from log4j-web. If the class is present, the shutdown hook is not used. This should solve the shutdown hook memory leak.

          Show
          Matt Sicker added a comment - In r1592429, I've added a check for the presence of a class from log4j-web. If the class is present, the shutdown hook is not used. This should solve the shutdown hook memory leak.
          Hide
          Matt Sicker added a comment -

          I began working on the initialization refactoring yesterday. Figuring out an appropriate way to use a sort of LoggerContext start-up strategy is where I'm at. I've got a branch in svn for that, too.

          Show
          Matt Sicker added a comment - I began working on the initialization refactoring yesterday. Figuring out an appropriate way to use a sort of LoggerContext start-up strategy is where I'm at. I've got a branch in svn for that, too.
          Remko Popma made changes -
          Resolution Won't Fix [ 2 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Remko Popma added a comment -

          Reopened. (We haven't found a solution yet, but that's not a good reason to close the issue.)

          Show
          Remko Popma added a comment - Reopened. (We haven't found a solution yet, but that's not a good reason to close the issue.)
          Hide
          Scott added a comment -

          Thanks for the effort you've invested thus far and its unfortunate this issue seems to be a difficult one. What is the next step? The LOG4J2-577 task that was designed to make this issue easier to resolve seems like a longer term project. What release is this task target toward? I fear that with this issue being marked as "RESOLVED" the issue may lose visibility and fall through the cracks. What is your feeling on the issue?

          Show
          Scott added a comment - Thanks for the effort you've invested thus far and its unfortunate this issue seems to be a difficult one. What is the next step? The LOG4J2-577 task that was designed to make this issue easier to resolve seems like a longer term project. What release is this task target toward? I fear that with this issue being marked as "RESOLVED" the issue may lose visibility and fall through the cracks. What is your feeling on the issue?
          Hide
          Matt Sicker added a comment -

          I tried using SoftReference, WeakReference, and PhantomReference (all with explicit use of enqueue() to mark for garbage collection), but the leak still remained. For some reason, there doesn't seem to be a way to unregister a shutdown hook thread in Tomcat.

          Show
          Matt Sicker added a comment - I tried using SoftReference, WeakReference, and PhantomReference (all with explicit use of enqueue() to mark for garbage collection), but the leak still remained. For some reason, there doesn't seem to be a way to unregister a shutdown hook thread in Tomcat.
          Hide
          Scott added a comment -

          Is there any new developments related to this issue? Were you able to find a more generic way to clean up the shutdown hook and resources dedicated to it (i.e. associated threads)? You mentioned the use of weak pointers...did that help at all?

          Show
          Scott added a comment - Is there any new developments related to this issue? Were you able to find a more generic way to clean up the shutdown hook and resources dedicated to it (i.e. associated threads)? You mentioned the use of weak pointers...did that help at all?
          Hide
          Matt Sicker added a comment -

          I tried that, but I didn't have any luck. Despite removing the shutdown hook, it would still leak the thread for some reason. Might be worth looking into more if I just did it wrong or something.

          Show
          Matt Sicker added a comment - I tried that, but I didn't have any luck. Despite removing the shutdown hook, it would still leak the thread for some reason. Might be worth looking into more if I just did it wrong or something.
          Hide
          Remko Popma added a comment -

          If I understand the above correctly, the shutdown hook is registered before log4j becomes aware that it is running in a web container. If that is the case, how about creating a new a method that unregisters the shutdown hook, and call this method as soon as log4j knows it is running in a web container?

          Show
          Remko Popma added a comment - If I understand the above correctly, the shutdown hook is registered before log4j becomes aware that it is running in a web container. If that is the case, how about creating a new a method that unregisters the shutdown hook, and call this method as soon as log4j knows it is running in a web container?
          Hide
          Matt Sicker added a comment -

          The runtime shutdown hook is just for non-container contexts. I think it would be better to use a more generic start/stop interface for dealing with LoggerContext objects. Selection of the proper context is already implemented through the ContextSelector interface. Initialization and destruction logic is handled in LoggerContext, but the hooks to start and stop them could be handled better.

          In the servlet context, there is no reason for the shutdown hook to be enabled. Due to the difficulty in overriding that shutdown hook in the current code is why I suggested we refactor that to be more generic to support more contexts.

          Show
          Matt Sicker added a comment - The runtime shutdown hook is just for non-container contexts. I think it would be better to use a more generic start/stop interface for dealing with LoggerContext objects. Selection of the proper context is already implemented through the ContextSelector interface. Initialization and destruction logic is handled in LoggerContext, but the hooks to start and stop them could be handled better. In the servlet context, there is no reason for the shutdown hook to be enabled. Due to the difficulty in overriding that shutdown hook in the current code is why I suggested we refactor that to be more generic to support more contexts.
          Hide
          Scott added a comment -

          Sounds promising. Can you explain why 2 shutdown hooks are needed? Is there some way tomcat (or servlet containers in general) can be shutdown where the ServletContextListener contextDestroyed is not called for each listener, but the Tomcat System.exit() event causes the additional shutdown hook to be called? In the simple use cases (stopping/unloading/redeploying, using the tomcat shutdown scripts, and even killing the Tomcat application from Visual VM) the contextDestroyed methods are called (verified via log4j2 log file output).

          Using a weak reference will likely help for this use-case, but is there a potential this will break other use-cases that are relying on a non-weak pointer?

          I created another issue for the JMX memory leak (LOG4J2-578) as per your earlier suggestion.

          Show
          Scott added a comment - Sounds promising. Can you explain why 2 shutdown hooks are needed? Is there some way tomcat (or servlet containers in general) can be shutdown where the ServletContextListener contextDestroyed is not called for each listener, but the Tomcat System.exit() event causes the additional shutdown hook to be called? In the simple use cases (stopping/unloading/redeploying, using the tomcat shutdown scripts, and even killing the Tomcat application from Visual VM) the contextDestroyed methods are called (verified via log4j2 log file output). Using a weak reference will likely help for this use-case, but is there a potential this will break other use-cases that are relying on a non-weak pointer? I created another issue for the JMX memory leak ( LOG4J2-578 ) as per your earlier suggestion.
          Hide
          Matt Sicker added a comment -

          In regard to the logback version, you can already do the same thing:

          import org.apache.logging.log4j.LogManager;
          import org.apache.logging.log4j.core.LoggerContext;
          // ...
          LoggerContext ctx = (LoggerContext) LogManager.getContext();
          ctx.stop();
          

          It's actually amusing how similar that is. If you use log4j in a servlet context, there's a ServletContextListener that starts and stops the LoggerContext on init and destroy respectively for the ServletContext. The memory leak was caused due to an additional shutdown hook thread being spawned that won't be called until Tomcat (or whatever server) calls System.exit() (clearly doesn't happen when an application is undeployed).

          Now if the shutdown hook is preventing the LoggerContext from being cleaned up, I think that could be fixed by using a weak reference in the shutdown thread so that it doesn't prevent clean up. I'm not so sure about the LoggerContextAdmin class. I'll take a look to see if I can find anything obvious, but I'm not experienced with the JMX standard.

          Show
          Matt Sicker added a comment - In regard to the logback version, you can already do the same thing: import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.core.LoggerContext; // ... LoggerContext ctx = (LoggerContext) LogManager.getContext(); ctx.stop(); It's actually amusing how similar that is. If you use log4j in a servlet context, there's a ServletContextListener that starts and stops the LoggerContext on init and destroy respectively for the ServletContext. The memory leak was caused due to an additional shutdown hook thread being spawned that won't be called until Tomcat (or whatever server) calls System.exit() (clearly doesn't happen when an application is undeployed). Now if the shutdown hook is preventing the LoggerContext from being cleaned up, I think that could be fixed by using a weak reference in the shutdown thread so that it doesn't prevent clean up. I'm not so sure about the LoggerContextAdmin class. I'll take a look to see if I can find anything obvious, but I'm not experienced with the JMX standard.
          Scott made changes -
          Labels memory_leak
          Scott made changes -
          Link This issue is part of LOG4J2-578 [ LOG4J2-578 ]
          Scott made changes -
          Environment Ubuntu 12.04
          Linux 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
          8 GB RAM

          java version "1.7.0_51"
          Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
          Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

          JAVA_OPTS=-Xmx1024m -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60 -Djava.net.preferIPv4Stack=true -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

          <log4j.version>2.0-rc1</log4j.version>
          log4j-api
          log4j-core
          log4j-jcl

          Spring webmvc 4.0.2.RELEASE application (simple hello world) deployed in tomcat7.0.52 container.
          Ubuntu 12.04
          Linux 3.2.0-58-generic x86_64 GNU/Linux
          8 GB RAM

          java version "1.7.0_51"
          Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
          Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

          JAVA_OPTS=-Xmx1024m -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60 -Djava.net.preferIPv4Stack=true -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

          <log4j.version>2.0-rc1</log4j.version>
          log4j-api
          log4j-core
          log4j-jcl

          Spring webmvc 4.0.2.RELEASE application (simple hello world) deployed in tomcat7.0.52 container.
          Scott made changes -
          Environment Ubuntu 12.04
          Linux bos-lpuv7 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
          8 GB RAM

          java version "1.7.0_51"
          Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
          Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

          JAVA_OPTS=-Xmx1024m -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60 -Djava.net.preferIPv4Stack=true -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

          <log4j.version>2.0-rc1</log4j.version>
          log4j-api
          log4j-core
          log4j-jcl

          Spring webmvc 4.0.2.RELEASE application (simple hello world) deployed in tomcat7.0.52 container.
          Ubuntu 12.04
          Linux 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
          8 GB RAM

          java version "1.7.0_51"
          Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
          Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

          JAVA_OPTS=-Xmx1024m -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60 -Djava.net.preferIPv4Stack=true -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

          <log4j.version>2.0-rc1</log4j.version>
          log4j-api
          log4j-core
          log4j-jcl

          Spring webmvc 4.0.2.RELEASE application (simple hello world) deployed in tomcat7.0.52 container.
          Hide
          Scott added a comment -

          I tried running tomcat under java 8 JVM and the problem still persists. I don't have a Mac OS X machine handy but I don't think it is likely that would make a difference.

          Tomcat Version: Apache Tomcat/7.0.52
          JVM Version: 1.8.0-b132	
          JVM Vendor: Oracle Corporation
          OS Name: Linux
          OS Version: 3.2.0-58-generic
          OS Architecture: amd64
          

          Let me know how you want to proceed. I know you have suggested opening another issue for the JMX memory leak but I will wait in-case you are still doing more analysis with the updated debugging instructions provided.

          If JMX is not needed for Log4j2 then a workaround is to disable it http://logging.apache.org/log4j/2.x/manual/jmx.html. I have verified that disabling JMX in combination with your proposed additional attribute to the log4j2.xml

          <Configuration shutdownHook="disable"> ... </Configuration>

          mitigates the memory issue for this use-case.

          Show
          Scott added a comment - I tried running tomcat under java 8 JVM and the problem still persists. I don't have a Mac OS X machine handy but I don't think it is likely that would make a difference. Tomcat Version: Apache Tomcat/7.0.52 JVM Version: 1.8.0-b132 JVM Vendor: Oracle Corporation OS Name: Linux OS Version: 3.2.0-58-generic OS Architecture: amd64 Let me know how you want to proceed. I know you have suggested opening another issue for the JMX memory leak but I will wait in-case you are still doing more analysis with the updated debugging instructions provided. If JMX is not needed for Log4j2 then a workaround is to disable it http://logging.apache.org/log4j/2.x/manual/jmx.html . I have verified that disabling JMX in combination with your proposed additional attribute to the log4j2.xml <Configuration shutdownHook= "disable" > ... </Configuration> mitigates the memory issue for this use-case.
          Hide
          Scott added a comment -

          It is always an option to create an OQL query (with javascript if supported) to answer the same question (find all gc roots exclude soft/weak references) the Eclipse Memory Analyzer can provide with a few button clicks. If you do go this route feel free to post your query here. Here are some references if you are interested:

          https://wiki.openitop.org/doku.php?id=oql:start
          http://docs.oracle.com/middleware/1212/jdev/OJDUG/optimizing_java_projects.htm
          https://visualvm.java.net/oqlhelp.html
          http://help.eclipse.org/kepler/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Ftasks%2Fqueryingheapobjects.html
          http://www.combodo.com/IMG/pdf/OQL_Reference.pdf

          Show
          Scott added a comment - It is always an option to create an OQL query (with javascript if supported) to answer the same question (find all gc roots exclude soft/weak references) the Eclipse Memory Analyzer can provide with a few button clicks. If you do go this route feel free to post your query here. Here are some references if you are interested: https://wiki.openitop.org/doku.php?id=oql:start http://docs.oracle.com/middleware/1212/jdev/OJDUG/optimizing_java_projects.htm https://visualvm.java.net/oqlhelp.html http://help.eclipse.org/kepler/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Ftasks%2Fqueryingheapobjects.html http://www.combodo.com/IMG/pdf/OQL_Reference.pdf
          Hide
          Scott added a comment -

          I have minimal experience with YourKit (no license), and slightly more with Visual VM. Below are some instructions on how to detect if there is a memory leak. My experience with these profiler tools is that it more difficult to actually trace the cause of the memory leak. This is where Eclipse Memory Analyzer ( https://www.eclipse.org/mat/ ) really shines (see other set of instructions below). Regardless if you like eclipse or not the filtering, searching, and object relationships (particularly to GC roots) provided by this plugin are very helpful.

          Profiler instructions (visual VM and should work for YourKit):
          1) Load the test webapp into your servlet container (tomcat).
          2) Find your servlet container in your favorite profiler.
          3) Take a heap dump.
          4) In the heap dump look for all instances of class "org.apache.catalina.loader.WebappClassLoader". There should be 1 instance of this class for each webapp that is loaded (started in tomcat's case). Inspect the "contextName" of each (assuming there is more than 1) instance until it matches your test application name.
          5) Stop your test webapp.
          6) Take a heap dump.
          7) There should now be 1 less instance of class "org.apache.catalina.loader.WebappClassLoader". I am claiming there is not 1 less and this is the memory leak.
          8) This is the step where YourKit and Visual VM where not so helpful...You want to find all non-weak/non-soft references from this object to all GC roots. For example Visual VM in the Out References you can right click "this" and "Show Nearest GC Root" but this does not yield very much context (often givens something general like a Task Thread). If you have a way to do this with Your Kit or another profiler tool please let me know.

          Eclipse Memory Analyzer (Eclipse Kepler) Instructions:
          1) Take a heap dump (File->New->Other->Other->Heap Dump)
          2) Select your application (Description=org.apache.catalina.startup.Boostrap start).
          3) Select anything but "Re-open previously run reports" (i.e. "Leak Suspects Report")
          4) Click the "Open Dominator Tree for entire heap" (third button from left in heap report)
          5) Search for "org.apache.catalina.loader.WebappClassLoader"
          6) Inspect each instance to find the instance which references your web application (look at strings to find paths to your JARs)
          7) Trace this instance back to GC roots (Right click->Path to GC Roots->exclude weak/soft references).
          8) In this case you should see an instance of class "org.apache.logging.log4j.core.jmx.LoggerContextAdmin" which is holding on to a reference.

          Show
          Scott added a comment - I have minimal experience with YourKit (no license), and slightly more with Visual VM. Below are some instructions on how to detect if there is a memory leak. My experience with these profiler tools is that it more difficult to actually trace the cause of the memory leak. This is where Eclipse Memory Analyzer ( https://www.eclipse.org/mat/ ) really shines (see other set of instructions below). Regardless if you like eclipse or not the filtering, searching, and object relationships (particularly to GC roots) provided by this plugin are very helpful. Profiler instructions (visual VM and should work for YourKit): 1) Load the test webapp into your servlet container (tomcat). 2) Find your servlet container in your favorite profiler. 3) Take a heap dump. 4) In the heap dump look for all instances of class "org.apache.catalina.loader.WebappClassLoader". There should be 1 instance of this class for each webapp that is loaded (started in tomcat's case). Inspect the "contextName" of each (assuming there is more than 1) instance until it matches your test application name. 5) Stop your test webapp. 6) Take a heap dump. 7) There should now be 1 less instance of class "org.apache.catalina.loader.WebappClassLoader". I am claiming there is not 1 less and this is the memory leak. 8) This is the step where YourKit and Visual VM where not so helpful...You want to find all non-weak/non-soft references from this object to all GC roots. For example Visual VM in the Out References you can right click "this" and "Show Nearest GC Root" but this does not yield very much context (often givens something general like a Task Thread). If you have a way to do this with Your Kit or another profiler tool please let me know. Eclipse Memory Analyzer (Eclipse Kepler) Instructions: 1) Take a heap dump (File->New->Other->Other->Heap Dump) 2) Select your application (Description=org.apache.catalina.startup.Boostrap start). 3) Select anything but "Re-open previously run reports" (i.e. "Leak Suspects Report") 4) Click the "Open Dominator Tree for entire heap" (third button from left in heap report) 5) Search for "org.apache.catalina.loader.WebappClassLoader" 6) Inspect each instance to find the instance which references your web application (look at strings to find paths to your JARs) 7) Trace this instance back to GC roots (Right click->Path to GC Roots->exclude weak/soft references). 8) In this case you should see an instance of class "org.apache.logging.log4j.core.jmx.LoggerContextAdmin" which is holding on to a reference.
          Hide
          Matt Sicker added a comment -

          As to the Tomcat thing, I should probably specify my environment, too.

          Server version: Apache Tomcat/7.0.52
          Server built:   Feb 13 2014 10:24:25
          Server number:  7.0.52.0
          OS Name:        Mac OS X
          OS Version:     10.8.5
          Architecture:   x86_64
          JVM Version:    1.8.0-b132
          JVM Vendor:     Oracle Corporation
          
          Show
          Matt Sicker added a comment - As to the Tomcat thing, I should probably specify my environment, too. Server version: Apache Tomcat/7.0.52 Server built: Feb 13 2014 10:24:25 Server number: 7.0.52.0 OS Name: Mac OS X OS Version: 10.8.5 Architecture: x86_64 JVM Version: 1.8.0-b132 JVM Vendor: Oracle Corporation
          Hide
          Matt Sicker added a comment -

          Here's a link to the refactoring. The JMX memory leak sounds like a separate issue. I suppose you could either edit this one or create a new one for that. Any advice on detecting the JMX leak might be helpful. I've got YourKit Java Profiler to use, but if you know of a better tool for finding leaks, I'm open to suggestions.

          Show
          Matt Sicker added a comment - Here's a link to the refactoring. The JMX memory leak sounds like a separate issue. I suppose you could either edit this one or create a new one for that. Any advice on detecting the JMX leak might be helpful. I've got YourKit Java Profiler to use, but if you know of a better tool for finding leaks, I'm open to suggestions.
          Matt Sicker made changes -
          Link This issue relates to LOG4J2-577 [ LOG4J2-577 ]
          Hide
          Scott added a comment - - edited

          When you say: "Tomact wasn't finding that other memory leak" do you mean the "Find Leaks" feature provided by tomcat ( http://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html#Finding_memory_leaks )? For my test case this feature was detecting there was a leak (I can provide heap dumps if necessary, are there any notable differences in our environments). This tomcat feature is nice but I'm not sure if tomcat detecting the leak or not is relevant (tomcat's memory leak detection functionality may not be guaranteed to detect all leaks). If you are just looking for ways to detect or inspect the memory leak I can suggest alternatives. It sounds like you have 1 case covered with the <Configuration> node modification, but I'm not confident this covers the whole issue. Could we leave this issue open and/or re-assign (to log4j2 JMX SME) if the new restructuring issue you are suggesting is not designed to capture/resolve this entire issue?

          Creating a separate issue for restructuring is fine as long as we link (in jira, and in description) the two issues together in such a way that the new issue must address the leak in the process of restructuring.

          Show
          Scott added a comment - - edited When you say: "Tomact wasn't finding that other memory leak" do you mean the "Find Leaks" feature provided by tomcat ( http://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html#Finding_memory_leaks )? For my test case this feature was detecting there was a leak (I can provide heap dumps if necessary, are there any notable differences in our environments). This tomcat feature is nice but I'm not sure if tomcat detecting the leak or not is relevant (tomcat's memory leak detection functionality may not be guaranteed to detect all leaks). If you are just looking for ways to detect or inspect the memory leak I can suggest alternatives. It sounds like you have 1 case covered with the <Configuration> node modification, but I'm not confident this covers the whole issue. Could we leave this issue open and/or re-assign (to log4j2 JMX SME) if the new restructuring issue you are suggesting is not designed to capture/resolve this entire issue? Creating a separate issue for restructuring is fine as long as we link (in jira, and in description) the two issues together in such a way that the new issue must address the leak in the process of restructuring.
          Hide
          Matt Sicker added a comment -

          Tomcat wasn't finding that other memory leak, so I'm not sure what to do about that. In regards to the no-fix, that's mainly because I'd like a separate issue raised for refactoring the initialization/destruction code to make this far more automatic. The current problem is the different code paths for JNDI versus an arbitrary ContextSelector. This actually relates a bit to how I'd like to see things get more modular to allow for more use case scenarios to be properly supported (e.g., OSGi, other container/module frameworks, Jigsaw in Java 9, etc).

          I've added a "big red" warning to the webapp.html documentation page that indicates you need to set shutdownHook="disable" to <Configuration/> (so yes, exactly as you said). I'm not familiar with the JMX code, so that might need to be looked at by someone else.

          In fact, I'll create a new issue for this because it's a separate but related issue.

          Show
          Matt Sicker added a comment - Tomcat wasn't finding that other memory leak, so I'm not sure what to do about that. In regards to the no-fix, that's mainly because I'd like a separate issue raised for refactoring the initialization/destruction code to make this far more automatic. The current problem is the different code paths for JNDI versus an arbitrary ContextSelector. This actually relates a bit to how I'd like to see things get more modular to allow for more use case scenarios to be properly supported (e.g., OSGi, other container/module frameworks, Jigsaw in Java 9, etc). I've added a "big red" warning to the webapp.html documentation page that indicates you need to set shutdownHook="disable" to <Configuration/> (so yes, exactly as you said). I'm not familiar with the JMX code, so that might need to be looked at by someone else. In fact, I'll create a new issue for this because it's a separate but related issue.
          Hide
          Scott added a comment - - edited

          Thank you for looking into this Matt. I had a feeling the solution to this issue may be constrained due to existing design and hooks w.r.t to servlet containers. It is unfortunate detection can not be done prior to adding the hook as Remko suggested. Are there usage statistics on log4j/log4j2 to know whether webapp/servlet deployment is a common enough use case to avoid a "no-fix" resolution? If the the proposed workaround (*see followup) does not work then I believe application developers are forced to use logback for this use-case (need to stop/redeploy/undeploy webapp). The difficult part for application developers is that the leak will likely go undetected until disaster strikes (PermGen exhausted), and may be difficult to diagnose for some developers in the webapp/servlet use-case.

          Given the current state of affairs it may be beneficial to reference the logback code base. They provide an accessor to the LoggerContext for which the stop method can be invoked (http://logback.qos.ch/manual/configuration.html#stopContext) to explicitly cleanup. More importantly they have figured out a way to either not start certain aspects of the infrastructure or clean up without explicit configuration or even explicitly closing the LoggerContext as described in their documentation (see code attached which uses logback-classic but does not close LoggerContext).

          *Can you clarify the proposed workaround? Is the workaround to add an attribute to the "log4j2.xml" <Configuration> element which is 'shutdownHook="disable"'? Is there anything else that needs to be done as I am still seeing a memory leak but with 1 layer of referencing removed. To clarify an instance of "org.apache.logging.log4j.core.jmx.LoggerContextAdmin" classloader is now the only class which still has hard references that remain (to "org.apache.catalina.loader.WebappClassLoader") and is preventing GC from cleaning up resources.

          Also can you provide a link to the updated documentation? If not already modified/clarified the "Web Applications and JSPs" documentation (http://logging.apache.org/log4j/2.x/manual/webapp.html#Servlet-3.0) states that log4j should "just work" w.r.t to deploying and shutting down.

          Show
          Scott added a comment - - edited Thank you for looking into this Matt. I had a feeling the solution to this issue may be constrained due to existing design and hooks w.r.t to servlet containers. It is unfortunate detection can not be done prior to adding the hook as Remko suggested. Are there usage statistics on log4j/log4j2 to know whether webapp/servlet deployment is a common enough use case to avoid a "no-fix" resolution? If the the proposed workaround (*see followup) does not work then I believe application developers are forced to use logback for this use-case (need to stop/redeploy/undeploy webapp). The difficult part for application developers is that the leak will likely go undetected until disaster strikes (PermGen exhausted), and may be difficult to diagnose for some developers in the webapp/servlet use-case. Given the current state of affairs it may be beneficial to reference the logback code base. They provide an accessor to the LoggerContext for which the stop method can be invoked ( http://logback.qos.ch/manual/configuration.html#stopContext ) to explicitly cleanup. More importantly they have figured out a way to either not start certain aspects of the infrastructure or clean up without explicit configuration or even explicitly closing the LoggerContext as described in their documentation (see code attached which uses logback-classic but does not close LoggerContext). *Can you clarify the proposed workaround? Is the workaround to add an attribute to the "log4j2.xml" <Configuration> element which is 'shutdownHook="disable"'? Is there anything else that needs to be done as I am still seeing a memory leak but with 1 layer of referencing removed. To clarify an instance of "org.apache.logging.log4j.core.jmx.LoggerContextAdmin" classloader is now the only class which still has hard references that remain (to "org.apache.catalina.loader.WebappClassLoader") and is preventing GC from cleaning up resources. Also can you provide a link to the updated documentation? If not already modified/clarified the "Web Applications and JSPs" documentation ( http://logging.apache.org/log4j/2.x/manual/webapp.html#Servlet-3.0 ) states that log4j should "just work" w.r.t to deploying and shutting down.
          Hide
          Matt Sicker added a comment -

          Not without a bit of refactoring. Right now, when you use the JNDI context selector, that gives you a location to override the hook before going to the Status.STARTING state. In the non-JNDI case, following the call chain leaves you in Log4jContextFactory.getContext(lots of params) where the LoggerContext is started. The only hook existing here would be in the ContextSelector itself, but that's still specified by configuration.

          Show
          Matt Sicker added a comment - Not without a bit of refactoring. Right now, when you use the JNDI context selector, that gives you a location to override the hook before going to the Status.STARTING state. In the non-JNDI case, following the call chain leaves you in Log4jContextFactory.getContext(lots of params) where the LoggerContext is started. The only hook existing here would be in the ContextSelector itself, but that's still specified by configuration.
          Hide
          Remko Popma added a comment - - edited

          I haven't looked at the code, but would it be possible to detect if we're running in a web container before adding the shutdown hook and registering or not registering depending on that?

          I mean, if it is hard to do this cleanly, but it is important to do it, then we could use something ugly like setting a global flag in the web initialization logic...

          Show
          Remko Popma added a comment - - edited I haven't looked at the code, but would it be possible to detect if we're running in a web container before adding the shutdown hook and registering or not registering depending on that? I mean, if it is hard to do this cleanly, but it is important to do it, then we could use something ugly like setting a global flag in the web initialization logic...
          Matt Sicker made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 2.0-rc2 [ 12326292 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Matt Sicker added a comment -

          I've updated the documentation to reflect this setting. I tried working in code that could prevent the shutdown hook from being used, but that turned out to not be feasible based on the current flow of execution. Removing a shutdown hook after it's been added in a web container doesn't appear to work, and said shutdown hook really won't be called until Tomcat (or similar) itself is shutdown. We already clean up using ServletContextListener. The best I can do here is possibly call thread.interrupt(), but I'm not sure if that's even a good idea.

          Show
          Matt Sicker added a comment - I've updated the documentation to reflect this setting. I tried working in code that could prevent the shutdown hook from being used, but that turned out to not be feasible based on the current flow of execution. Removing a shutdown hook after it's been added in a web container doesn't appear to work, and said shutdown hook really won't be called until Tomcat (or similar) itself is shutdown. We already clean up using ServletContextListener. The best I can do here is possibly call thread.interrupt(), but I'm not sure if that's even a good idea.
          Hide
          Matt Sicker added a comment -

          I could add some logic to pre-disable the shutdown hook in o.a.l.l.c.LoggerContext that the web initialization code can use. I'll make a commit, and you can review that to see if it's too complicated to be worth it or not.

          Show
          Matt Sicker added a comment - I could add some logic to pre-disable the shutdown hook in o.a.l.l.c.LoggerContext that the web initialization code can use. I'll make a commit, and you can review that to see if it's too complicated to be worth it or not.
          Hide
          Gary Gregory added a comment -

          so add a big red warning in the docs?

          Show
          Gary Gregory added a comment - so add a big red warning in the docs?
          Hide
          Matt Sicker added a comment -

          Scratch that, I found the bug. This is a two-parter:

          1. The shutdown hook is enabled by default. In a web context, this should be disabled using the shutdownHook="disabled" attribute in <Configuration/> (or equivalent in JSON or YAML).
          2. This should be default behavior in a web context; otherwise, the shutdown hook thread will leak, and there doesn't seem to be any way to prevent that without doing server-specific cases (which is also a bad idea).
          Show
          Matt Sicker added a comment - Scratch that, I found the bug. This is a two-parter: The shutdown hook is enabled by default. In a web context, this should be disabled using the shutdownHook="disabled" attribute in <Configuration/> (or equivalent in JSON or YAML). This should be default behavior in a web context; otherwise, the shutdown hook thread will leak, and there doesn't seem to be any way to prevent that without doing server-specific cases (which is also a bad idea).
          Hide
          Matt Sicker added a comment -

          Well this is turning out to be a bit more complicated than I originally thought! Spawning off a shutdown hook thread in a web container is a bad idea due to problems like this, so I'm looking at alternative ways to shutdown in a web context. I see that we could use a ServletContextListener to shut down when the ServletContext is destroyed, but we'll need to make sure that's only done for the LoggerContext associated to that ServletContext so that if log4j is a server library, it doesn't shut down everyone. Seems like this could work. I think the JMX class that stays loaded is due to the shutdown thread staying behind which references the LoggerContext linked to the JMX class.

          Show
          Matt Sicker added a comment - Well this is turning out to be a bit more complicated than I originally thought! Spawning off a shutdown hook thread in a web container is a bad idea due to problems like this, so I'm looking at alternative ways to shutdown in a web context. I see that we could use a ServletContextListener to shut down when the ServletContext is destroyed, but we'll need to make sure that's only done for the LoggerContext associated to that ServletContext so that if log4j is a server library, it doesn't shut down everyone. Seems like this could work. I think the JMX class that stays loaded is due to the shutdown thread staying behind which references the LoggerContext linked to the JMX class.
          Matt Sicker made changes -
          Assignee Matt Sicker [ jvz ]
          Scott made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Scott made changes -
          Description Dynamically loading a JAR that uses log4j2 results in a memory leak when the JAR is unloaded. This can be observed in the while deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Dynamically loading a JAR that uses log4j2 results in a memory leak when the JAR is unloaded. This can be observed by deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Scott made changes -
          Description Dynamically loading a JAR that uses log4j2 results in a memory leak when then JAR is unloaded. This can be observed in the while deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Dynamically loading a JAR that uses log4j2 results in a memory leak when the JAR is unloaded. This can be observed in the while deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Scott made changes -
          Description The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Dynamically loading a JAR that uses log4j2 results in a memory leak when then JAR is unloaded. This can be observed in the while deploying a web application to tomcat7 and exercising the stop, undeploy, or redeploy actions. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Scott made changes -
          Summary Destroy Time Memory Leak Memory Leak
          Scott made changes -
          Summary Memory Leak Destroy Time Memory Leak
          Scott made changes -
          Description The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be caused by log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Scott made changes -
          Original Estimate 336h [ 1209600 ]
          Remaining Estimate 336h [ 1209600 ]
          Scott made changes -
          Fix Version/s 2.0-rc2 [ 12326292 ]
          Scott made changes -
          Labels memory_leak
          Scott made changes -
          Attachment spring_log4j2_memory_leak.tbz2 [ 12635547 ]
          Hide
          Scott added a comment -

          This is the code (see README) which demonstrates memory leak. The hprof which demonstrates the before/after memory leak state is too big (>10MB). This can be provided upon request.

          Show
          Scott added a comment - This is the code (see README) which demonstrates memory leak. The hprof which demonstrates the before/after memory leak state is too big (>10MB). This can be provided upon request.
          Scott made changes -
          Description The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible and I can provide more information if necessary. I don't see anywhere to attach a heap dump, but a sequence of these and simple hello world source code can be supplied upon request.
          The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using SLF4J (slf4j-api, jcl-over-slf4j) to logback-classic logging output is equivalent but all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)
          3)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible with a very simple spring hello world application. Code and/or heap dumps can be provided upon request.
          Scott made changes -
          Field Original Value New Value
          Description The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j leaves creates a memory leak. The memory leak is believed to be log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible and I can provide more information if necessary. I don't see anywhere to attach a heap dump, but a sequence of these and simple hello world source code can be supplied upon request.
          The tomcat7 stop, undeploy, or redeploy actions on a WAR which utilizes log4j can create a memory leak. The memory leak is believed to be log4j for the following reasons:
          1)Heap Dump reveals the classloader instance responsible for the WAR plugin (of type org.apache.catalina.loader.WebappClassLoader) has 2 non weak/soft reference which are of type (org.apache.logging.log4j.core.LoggerContext$ShutdownThread) and (org.apache.logging.log4j.core.jmx.LoggerContextAdmin) after the WAR has been stopped or undeployed.
          2)Using the SLF4J NOP logger implementation all memory is gc as expected (the org.apache.catalina.loader.WebappClassLoader which loaded the WAR is no longer referenced by any hard references)

          This may not be unique to 2.0rc-1 and I have seen similar behavior in previous 2.0 beta releases.

          This is reproducible and I can provide more information if necessary. I don't see anywhere to attach a heap dump, but a sequence of these and simple hello world source code can be supplied upon request.
          Scott created issue -

            People

            • Assignee:
              Matt Sicker
              Reporter:
              Scott
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development