[LOGCXX-322] Crashes on exit from multithreaded program using log4cxx

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.12.0
• Environment: Linux platform server product, compiled with g++ 4.3.x, running on multi-core IA32 and IA64 hardware
• Assignee: Curt Arnold
• Reporter: Ben Hockley
• Created: 19/Jan/09 12:36
• Updated: 11/Dec/21 16:00
• Resolved: 23/Mar/21 01:11

Description

On exit(0); (attempt at graceful process exit) we commonly see segmentation faults in:

• log4cxx::helpers::ObjectPtrBase()

• log4cxx::LogManager::getLoggerLS()

The cause

=======

We eventually traced this issue to the pervasive pattern of usage of singletons in the log4cxx code.

The log4cxx library uses the "Meyers" singleton pattern, first popularised in Scott Meyers "Effective C++" (Item 47 in the 2nd edition of that book):

Thing & getThingSingleton()

{ static Thing t; return t; }

For many years, the above pattern was considered "best practice" for using Singletons in C++ - and was generally safe for most popular compiler implementations and most applications.

Unfortunately, this recommendation is not actually guaranteed to be thread-safe for construction or destruction - something which is alluded to on Scott Meyers' own "Errata List for Effective C++, Second Edition"

as described here: http://www.aristeia.com/BookErrata/ec++2e-errata.html

The nub of the problem is that when a process calls "exit(0);" or similar, one thread will start running, in order, any user-registered "atexit" functions.

Along with these, the compiler will execute the (conceptually similar) compiler-registered functions which invoke the destructors of any static file or function scope objects (also in order - the reverse order to static object construction).

Unfortunately, other threads may still be in the process of running -and logging, perhaps using the static objects - during or after the execution of their destructors - thus opening the door to a bunch of potential SEGFAULTs.

Solutions

======

The solution is to introduce an (optional) new singleton lifetime management policy. Specifically, we need to make sure that any Singletons required for log4cxx logging to operate correctly (including, but not limited to, APRInitializer) are always available when required by any thread - including during process shutdown.

Andrei Alexandrescu goes into a lot of detail about this C++ design problem in chapter 6 of "Modern C++ design" and proposes two elegant solutions - a "Phoenix Singleton" or SingletonHolder class.

For complete thread-safety, we should also consider the startup/initialization race for each singleton (for that we'd need a multiple-locked singleton initialization pattern everywhere log4cxx creates a singleton).

We will need to address the singletons in:

• aprinitializer.cpp
• level.cpp
• logmanager.cpp

and perhaps elsewhere too.

An attempt at patching the above files to simply "leak" the crucial singleton objects seemed to prevent the crashes (but could, of course, cause unacceptable behaviour in certain embedded environments - or in environments using memory leak checkers like Boundshecker, Purify, Valgrind etc.)

Curt Arnold suggested (on the mailing list) "Probably the best approach is to try to isolate the singleton pattern into a preprocessor macro and then allow the user to select what singleton pattern they'd like to use." - something which sounds like a very wise idea.

Issue Links

• blocks: LOGCXX-363 log4cxx 0.10.1 release
• is cloned by: LOGCXX-338 Mutex exception
• is duplicated by: LOGCXX-289 In Solaris Sparc - Using log4cxx.0.10.0 Sample program is crashing on Exit.
• is duplicated by: LOGCXX-343 Logger::addAppender from DLL causes crash upon process exit
• is duplicated by: LOGCXX-352 crash on program exit
• is duplicated by: LOGCXX-338 Mutex exception
• is duplicated by: LOGCXX-449 crash on program exit when no logging output is produced
• is duplicated by: LOGCXX-344 Log4cxx crash when running multiple threads
• is related to: LOGCXX-532 Static objects and deleting
• relates to: LOGCXX-352 crash on program exit
• relates to: LOGCXX-430 LogManager::getRootLogger is not thread-safe
• relates to: LOGCXX-394 Levels are not thread safe
• links to: GitHub Pull Request #55