Uploaded image for project: 'Log4j 2'
  1. Log4j 2
  2. LOG4J2-1434

StringBuffer in ThreadLocal can cause excessive memory usage after large log messages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.1
    • Fix Version/s: 2.6.2
    • Component/s: None
    • Labels:
      None

      Description

      In an effort to speed up logging ThreadLocals have been introduced see LOG4J2-1125 however this does causes memory issues.

      The problem of the ThreadLocal occurs when threads are re-used which is an absolutely valid way of using java. For example an executor service can re-use threads as well as Jetty.

      Below I demonstrate a contrived example of the memory leak:

      int stringSize = 1024*1024*10; //~10MB maybe 20MB for UTF-16
              StringBuilder sb = new StringBuilder(stringSize); 
              for(int i = 0; i < stringSize; i++) {
                  sb.append('a' + i % 5);
              }
              
              String largeString = sb.toString();
              
              sb = null; //Let it be GC'ed
              ExecutorService es = Executors.newFixedThreadPool(100);
              final CountDownLatch countDownLatch = new CountDownLatch(100);
              for(int i = 0; i < 100; i++) {
                  es.execute(()-> {
                      //Log the big string to demonstrate the issue.
                      log.fatal(largeString);
                      
                      //Ensure we use all 100 of our threads by not releasing this thread yet.
                      countDownLatch.countDown();
                  }); 
                  
                  //We sleep for 2s so we more easily watch memory growth
                  Thread.sleep(2000);
              }
      

      I recommend that log4j2 immediately remove the ThreadLocal as a small gain in performance does not outweigh the problems associated with memory leaks. Finally other options for caching the StringBuilder with a ThreadLocal could be considered for example we might re-use StringBuilders that are no larger than 3k while removing the ones which are larger than 3k.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                remkop@yahoo.com Remko Popma
                Reporter:
                lukebutters7 Luke Butters
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: