[IMPALA-5884] Impala daemon crashes during attempted clean shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: supportability
• Target Version: Product Backlog
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: Lars Volker
• Created: 31/Aug/17 21:47
• Updated: 21/Jan/20 21:12
• Resolved: 21/Jan/20 21:12

Description

We have seen a number of different-looking crashes that all occur while the process is trying to shut down. In all these cases there's a thread with vm_exit() and JVM_Halt() in the stack. This is executed by the JVM when it is trying to do a clean shutdown of the process. For example, we see crashes like this:

Crash reason:  SIGSEGV
Crash address: 0x28
Process uptime: not available

Thread 587 (crashed)
 0  impalad!impala::ClientCacheHelper::ReleaseClient(void**) [client-cache.cc : 153 + 0x0]
    Found by: given as instruction pointer in context
 1  impalad!impala::ClientConnection<impala::ImpalaBackendClient>::~ClientConnection() [client-cache.h : 412 + 0x5]
    Found by: call frame info
 2  impalad!impala::DataStreamSender::Channel::TransmitDataHelper(impala::TRowBatch const*) [data-stream-sender.cc : 208 + 0x9]
    Found by: call frame info
 3  impalad!impala::DataStreamSender::Channel::TransmitData(int, impala::TRowBatch const*) [data-stream-sender.cc : 185 + 0x5]
    Found by: call frame info
 4  impalad!impala::ThreadPool<impala::TRowBatch*>::WorkerThread(int) [function_template.hpp : 767 + 0x10]
    Found by: call frame info
 5  impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) [function_template.hpp : 767 + 0x7]
    Found by: call frame info
 6  impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() [bind.hpp : 457 + 0x6]
    Found by: call frame info
 7  impalad!thread_proxy + 0xda
    Found by: call frame info
 8  libpthread-2.12.so + 0x7a51
    Found by: call frame info
 9  libc-2.12.so + 0xe893d
    Found by: stack scanning

The other threads doing the shutdown are

 0  libpthread-2.12.so + 0xb68c
    Found by: given as instruction pointer in context
 1  libjvm.so!Monitor::IWait(Thread*, long) + 0xf7
    Found by: previous frame's frame pointer
 2  libjvm.so!Monitor::wait(bool, long, bool) + 0x2d6
    Found by: call frame info
 3  libjvm.so!VMThread::execute(VM_Operation*) + 0x324
    Found by: call frame info
 4  libjvm.so!vm_exit(int) + 0x83
    Found by: call frame info
 5  libjvm.so!JVM_Halt + 0xba
    Found by: call frame info
 6  0x7f8ac53cda34
    Found by: call frame info
 7  0x7f8ac53be2bd
    Found by: previous frame's frame pointer
 8  0x7f8ac53be2bd
    Found by: previous frame's frame pointer
 9  0x7f8ac53be2bd
    Found by: previous frame's frame pointer
10  0x7f8ac53be302
    Found by: previous frame's frame pointer
11  0x7f8ac6fe1104
    Found by: previous frame's frame pointer
12  libjvm.so!JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x1056
    Found by: previous frame's frame pointer
13  libjvm.so!JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) + 0x321
    Found by: call frame info
14  libjvm.so!JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*) + 0x47
    Found by: call frame info
15  libjvm.so!thread_entry(JavaThread*, Thread*) + 0xa0
    Found by: call frame info
16  libjvm.so!JavaThread::thread_main_inner() + 0x103
    Found by: call frame info
17  libjvm.so!JavaThread::run() + 0x11c
    Found by: call frame info
18  libjvm.so!java_start(Thread*) + 0x108
    Found by: call frame info
19  libpthread-2.12.so + 0x7aa1
    Found by: call frame info
20  libc-2.12.so + 0xe893d
    Found by: stack scanning


Thread 29
 0  libpthread-2.12.so + 0xba5e
    Found by: given as instruction pointer in context
 1  libjvm.so!Monitor::IWait(Thread*, long) + 0x44e
    Found by: previous frame's frame pointer
 2  libjvm.so!Monitor::wait(bool, long, bool) + 0x256
    Found by: call frame info
 3  libjvm.so!VM_Exit::wait_for_threads_in_native_to_block() + 0x118
    Found by: call frame info
 4  libjvm.so!VM_Exit::doit() + 0x16
    Found by: call frame info
 5  libjvm.so!VM_Operation::evaluate() + 0x55
    Found by: call frame info
 6  libjvm.so!VMThread::evaluate_operation(VM_Operation*) + 0xba
    Found by: call frame info
 7  libjvm.so!VMThread::loop() + 0x1ce
    Found by: call frame info
 8  libjvm.so!VMThread::run() + 0x70
    Found by: call frame info
 9  libjvm.so!java_start(Thread*) + 0x108
    Found by: call frame info
10  libpthread-2.12.so + 0x7aa1
    Found by: call frame info
11  libc-2.12.so + 0xe893d
    Found by: stack scanning

Issue Links

• is duplicated by: IMPALA-5874 Segfault in TBinaryProtocolT<apache::thrift::transport::TTransport>::readI32(int&) [breakpad?]
• is duplicated by: IMPALA-5875 Segmentation fault in libjvm.so!OopMapSet::find_map_at_offset(int) [breakpad?]
• is duplicated by: IMPALA-5876 Segmentation fault in HdfsAvroScanner::MaterializeTuple(impala::AvroSchemaElement const&, impala::MemPool*, unsigned char**, unsigned char*, impala::Tuple*)
• is duplicated by: IMPALA-5877 Segmentation fault in libjvm.so!VM_Exit::wait_for_threads_in_native_to_block() [breakpad?]
• is duplicated by: IMPALA-5879 Segmentation fault in llvm::Instruction::Instruction(llvm::Type*, unsigned int, llvm::Use*, unsigned int, llvm::Instruction*) [breakpad?]
• is duplicated by: IMPALA-5880 Segmentation fault in impala::HdfsScanNodeBase::Prepare(impala::RuntimeState*)
• is duplicated by: IMPALA-5882 Segmentation fault in MemPool::FindChunk(long, bool)
• is duplicated by: IMPALA-5883 Segmentation fault in ClientCacheHelper::ReleaseClient(void**) [breakpad?]
• is related to: IMPALA-6263 Assert hit during service restart Mutex.cpp:130: apache::thrift::concurrency::Mutex::impl::~impl(): Assertion `ret == 0' failed
• relates to: IMPALA-5891 PeriodicCounterUpdater should not rely on static initialisation and destruction order