Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2237

allow TimeLimitedCollector timer thread to be shutdown

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.4, 2.4.1, 2.9, 2.9.1, 3.0
    • Fix Version/s: None
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      When using Tomcat 6 and Solr 1.3 (with Lucene 2.4) we found that if we caused Tomcat to reload our .war files a number of times, we would eventually see PermGen memory errors where the JVM' s GC reported that all "permanent generation" memory had been consumed and none could be freed. This turns out to be a fairly common issue when using Tomcat's autoDeploy feature (or similar features of other application servers). See, for example:

      http://ampedandwired.dreamhosters.com/2008/05/09/causes-of-java-permgen-memory-leaks/
      http://cornelcreanga.com/2009/02/how-to-prevent-memory-leaks-when-reloading-web-applications/
      http://www.samaxes.com/2007/10/classloader-leaks-and-permgen-space/
      http://blogs.sun.com/fkieviet/entry/how_to_fix_the_dreaded

      My understanding of the issue is that when reloading a webapp, Tomcat starts by releasing all of its references to the ClassLoader used to load the previous version of the application. Then it creates a new ClassLoader which reloads the application. The old ClassLoader and old version of the app are left to the garbage collector to be cleaned up. However, if the app itself hold references to the ClassLoader, the GC may not be able to ascertain that the ClassLoader is truly unused, in which case, it and the entire old version of app remain in memory. If one causes a sufficient number of app reloads, eventually PermGen space is exhausted.

      The particular issue we had with Solr and Lucene was that Lucene's TimeLimitedCollector creates a thread which is not shut down anywhere; this in turn seems to prevent Tomcat from unloading the ClassLoader. To solve this I applied a minor patch to TimeLimitedCollector which adds a flag variable controlling the timer thread's loop and some methods to set it so the thread will exit.

      The stopThread() method can then be called by an application such as Solr from a class registered as a servlet context listener; when the server is unloading the application the listener will execute and in turn stop the timer thread. My testing during multiple reloads of solr.war with and without these patches indicates that without them, we consistently get PermGen errors, and with them, once the PermGen is nearly exhausted (which may take a lot of reloads, e.g., 10-15!), the GC is able to free space and no PermGen errors occur.

        Attachments

        1. LUCENE-2237-2_4.patch
          1 kB
          Chris Darroch

          Issue Links

            Activity

              People

              • Assignee:
                simonw Simon Willnauer
                Reporter:
                cdarroch Chris Darroch
              • Votes:
                2 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: