[LOGCXX-322] Crashes on exit from multithreaded program using log4cxx - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 0.10.0
Fix Version/s: 0.12.0
Component/s: None
Labels:
None
Environment:
Linux platform server product, compiled with g++ 4.3.x, running on multi-core IA32 and IA64 hardware

Description

On exit(0); (attempt at graceful process exit) we commonly see segmentation faults in:

log4cxx::helpers::ObjectPtrBase()

log4cxx::LogManager::getLoggerLS()

The cause

=======

We eventually traced this issue to the pervasive pattern of usage of singletons in the log4cxx code.

The log4cxx library uses the "Meyers" singleton pattern, first popularised in Scott Meyers "Effective C++" (Item 47 in the 2nd edition of that book):

Thing & getThingSingleton()

{ static Thing t; return t; }

For many years, the above pattern was considered "best practice" for using Singletons in C++ - and was generally safe for most popular compiler implementations and most applications.

Unfortunately, this recommendation is not actually guaranteed to be thread-safe for construction or destruction - something which is alluded to on Scott Meyers' own "Errata List for Effective C++, Second Edition"

as described here: http://www.aristeia.com/BookErrata/ec++2e-errata.html

The nub of the problem is that when a process calls "exit(0);" or similar, one thread will start running, in order, any user-registered "atexit" functions.

Along with these, the compiler will execute the (conceptually similar) compiler-registered functions which invoke the destructors of any static file or function scope objects (also in order - the reverse order to static object construction).

Unfortunately, other threads may still be in the process of running -and logging, perhaps using the static objects - during or after the execution of their destructors - thus opening the door to a bunch of potential SEGFAULTs.

Solutions

======

The solution is to introduce an (optional) new singleton lifetime management policy. Specifically, we need to make sure that any Singletons required for log4cxx logging to operate correctly (including, but not limited to, APRInitializer) are always available when required by any thread - including during process shutdown.

Andrei Alexandrescu goes into a lot of detail about this C++ design problem in chapter 6 of "Modern C++ design" and proposes two elegant solutions - a "Phoenix Singleton" or SingletonHolder class.

For complete thread-safety, we should also consider the startup/initialization race for each singleton (for that we'd need a multiple-locked singleton initialization pattern everywhere log4cxx creates a singleton).

We will need to address the singletons in:

aprinitializer.cpp
level.cpp
logmanager.cpp

and perhaps elsewhere too.

An attempt at patching the above files to simply "leak" the crucial singleton objects seemed to prevent the crashes (but could, of course, cause unacceptable behaviour in certain embedded environments - or in environments using memory leak checkers like Boundshecker, Purify, Valgrind etc.)

Curt Arnold suggested (on the mailing list) "Probably the best approach is to try to isolate the singleton pattern into a preprocessor macro and then allow the user to select what singleton pattern they'd like to use." - something which sounds like a very wise idea.