[IMPALA-5891] PeriodicCounterUpdater should not rely on static initialisation and destruction order

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Target Version: Impala 2.11.0
• Epic Color: ghx-label-2
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 01/Sep/17 21:30
• Updated: 05/Sep/17 21:41
• Resolved: 05/Sep/17 21:41

Description

The lifecycle of PeriodicCounterUpdater is weird and depends on static initialisers and destructors running.

Its constructor spawns a thread and its destructor tests down data structures in some probably unsafe order.

I've seen a crash in the code which could be caused by the data structures being torn down while the thread is still running:

Thread 1
 0  libpthread-2.12.so + 0xe2e4
    Found by: given as instruction pointer in context
 1  libpthread-2.12.so + 0x9588
    Found by: stack scanning
 2  impalad!google_breakpad::ExceptionHandler::SignalHandler(int, siginfo*, void*) + 0x23
    Found by: stack scanning
 3  libjvm.so!os::Linux::chained_handler(int, siginfo*, void*) + 0x112
    Found by: call frame info
 4  libjvm.so!JVM_handle_linux_signal + 0xb6
    Found by: call frame info
 5  libjvm.so!signalHandler(int, siginfo*, void*) + 0x43
    Found by: call frame info
 6  libpthread-2.12.so + 0xf790
    Found by: call frame info
 7  impalad!impala::PeriodicCounterUpdater::UpdateLoop() [function_template.hpp : 767 + 0x4]
    Found by: stack scanning
 8  0xffff
    Found by: call frame info

Instead of the static instance object we should just allocate the state on the heap.

Issue Links

• is related to: IMPALA-5884 Impala daemon crashes during attempted clean shutdown