Uploaded image for project: 'Log4cxx'
  1. Log4cxx
  2. LOGCXX-322

Crashes on exit from multithreaded program using log4cxx

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Environment:
      Linux platform server product, compiled with g++ 4.3.x, running on multi-core IA32 and IA64 hardware

      Description

      On exit(0); (attempt at graceful process exit) we commonly see segmentation faults in:

      • log4cxx::helpers::ObjectPtrBase()
      • log4cxx::LogManager::getLoggerLS()

      The cause

      =======

      We eventually traced this issue to the pervasive pattern of usage of singletons in the log4cxx code.

      The log4cxx library uses the "Meyers" singleton pattern, first popularised in Scott Meyers "Effective C++" (Item 47 in the 2nd edition of that book):

      Thing & getThingSingleton()

      { static Thing t; return t; }

      For many years, the above pattern was considered "best practice" for using Singletons in C++ - and was generally safe for most popular compiler implementations and most applications.

      Unfortunately, this recommendation is not actually guaranteed to be thread-safe for construction or destruction - something which is alluded to on Scott Meyers' own "Errata List for Effective C++, Second Edition"

      as described here: http://www.aristeia.com/BookErrata/ec++2e-errata.html

      The nub of the problem is that when a process calls "exit(0);" or similar, one thread will start running, in order, any user-registered "atexit" functions.

      Along with these, the compiler will execute the (conceptually similar) compiler-registered functions which invoke the destructors of any static file or function scope objects (also in order - the reverse order to static object construction).

      Unfortunately, other threads may still be in the process of running -and logging, perhaps using the static objects - during or after the execution of their destructors - thus opening the door to a bunch of potential SEGFAULTs.

      Solutions

      ======

      The solution is to introduce an (optional) new singleton lifetime management policy. Specifically, we need to make sure that any Singletons required for log4cxx logging to operate correctly (including, but not limited to, APRInitializer) are always available when required by any thread - including during process shutdown.

      Andrei Alexandrescu goes into a lot of detail about this C++ design problem in chapter 6 of "Modern C++ design" and proposes two elegant solutions - a "Phoenix Singleton" or SingletonHolder class.

      For complete thread-safety, we should also consider the startup/initialization race for each singleton (for that we'd need a multiple-locked singleton initialization pattern everywhere log4cxx creates a singleton).

      We will need to address the singletons in:

      • aprinitializer.cpp
      • level.cpp
      • logmanager.cpp

      and perhaps elsewhere too.

      An attempt at patching the above files to simply "leak" the crucial singleton objects seemed to prevent the crashes (but could, of course, cause unacceptable behaviour in certain embedded environments - or in environments using memory leak checkers like Boundshecker, Purify, Valgrind etc.)

      Curt Arnold suggested (on the mailing list) "Probably the best approach is to try to isolate the singleton pattern into a preprocessor macro and then allow the user to select what singleton pattern they'd like to use." - something which sounds like a very wise idea.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              carnold@apache.org Curt Arnold
              Reporter:
              rhosyn Ben Hockley

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment