Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
Impala 2.9.0
-
None
-
ghx-label-6
Description
We have seen a number of different-looking crashes that all occur while the process is trying to shut down. In all these cases there's a thread with vm_exit() and JVM_Halt() in the stack. This is executed by the JVM when it is trying to do a clean shutdown of the process. For example, we see crashes like this:
Crash reason: SIGSEGV Crash address: 0x28 Process uptime: not available Thread 587 (crashed) 0 impalad!impala::ClientCacheHelper::ReleaseClient(void**) [client-cache.cc : 153 + 0x0] Found by: given as instruction pointer in context 1 impalad!impala::ClientConnection<impala::ImpalaBackendClient>::~ClientConnection() [client-cache.h : 412 + 0x5] Found by: call frame info 2 impalad!impala::DataStreamSender::Channel::TransmitDataHelper(impala::TRowBatch const*) [data-stream-sender.cc : 208 + 0x9] Found by: call frame info 3 impalad!impala::DataStreamSender::Channel::TransmitData(int, impala::TRowBatch const*) [data-stream-sender.cc : 185 + 0x5] Found by: call frame info 4 impalad!impala::ThreadPool<impala::TRowBatch*>::WorkerThread(int) [function_template.hpp : 767 + 0x10] Found by: call frame info 5 impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) [function_template.hpp : 767 + 0x7] Found by: call frame info 6 impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() [bind.hpp : 457 + 0x6] Found by: call frame info 7 impalad!thread_proxy + 0xda Found by: call frame info 8 libpthread-2.12.so + 0x7a51 Found by: call frame info 9 libc-2.12.so + 0xe893d Found by: stack scanning
The other threads doing the shutdown are
0 libpthread-2.12.so + 0xb68c Found by: given as instruction pointer in context 1 libjvm.so!Monitor::IWait(Thread*, long) + 0xf7 Found by: previous frame's frame pointer 2 libjvm.so!Monitor::wait(bool, long, bool) + 0x2d6 Found by: call frame info 3 libjvm.so!VMThread::execute(VM_Operation*) + 0x324 Found by: call frame info 4 libjvm.so!vm_exit(int) + 0x83 Found by: call frame info 5 libjvm.so!JVM_Halt + 0xba Found by: call frame info 6 0x7f8ac53cda34 Found by: call frame info 7 0x7f8ac53be2bd Found by: previous frame's frame pointer 8 0x7f8ac53be2bd Found by: previous frame's frame pointer 9 0x7f8ac53be2bd Found by: previous frame's frame pointer 10 0x7f8ac53be302 Found by: previous frame's frame pointer 11 0x7f8ac6fe1104 Found by: previous frame's frame pointer 12 libjvm.so!JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x1056 Found by: previous frame's frame pointer 13 libjvm.so!JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) + 0x321 Found by: call frame info 14 libjvm.so!JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*) + 0x47 Found by: call frame info 15 libjvm.so!thread_entry(JavaThread*, Thread*) + 0xa0 Found by: call frame info 16 libjvm.so!JavaThread::thread_main_inner() + 0x103 Found by: call frame info 17 libjvm.so!JavaThread::run() + 0x11c Found by: call frame info 18 libjvm.so!java_start(Thread*) + 0x108 Found by: call frame info 19 libpthread-2.12.so + 0x7aa1 Found by: call frame info 20 libc-2.12.so + 0xe893d Found by: stack scanning Thread 29 0 libpthread-2.12.so + 0xba5e Found by: given as instruction pointer in context 1 libjvm.so!Monitor::IWait(Thread*, long) + 0x44e Found by: previous frame's frame pointer 2 libjvm.so!Monitor::wait(bool, long, bool) + 0x256 Found by: call frame info 3 libjvm.so!VM_Exit::wait_for_threads_in_native_to_block() + 0x118 Found by: call frame info 4 libjvm.so!VM_Exit::doit() + 0x16 Found by: call frame info 5 libjvm.so!VM_Operation::evaluate() + 0x55 Found by: call frame info 6 libjvm.so!VMThread::evaluate_operation(VM_Operation*) + 0xba Found by: call frame info 7 libjvm.so!VMThread::loop() + 0x1ce Found by: call frame info 8 libjvm.so!VMThread::run() + 0x70 Found by: call frame info 9 libjvm.so!java_start(Thread*) + 0x108 Found by: call frame info 10 libpthread-2.12.so + 0x7aa1 Found by: call frame info 11 libc-2.12.so + 0xe893d Found by: stack scanning
Attachments
Attachments
Issue Links
- is duplicated by
-
IMPALA-5874 Segfault in TBinaryProtocolT<apache::thrift::transport::TTransport>::readI32(int&) [breakpad?]
- Resolved
-
IMPALA-5875 Segmentation fault in libjvm.so!OopMapSet::find_map_at_offset(int) [breakpad?]
- Resolved
-
IMPALA-5876 Segmentation fault in HdfsAvroScanner::MaterializeTuple(impala::AvroSchemaElement const&, impala::MemPool*, unsigned char**, unsigned char*, impala::Tuple*)
- Resolved
-
IMPALA-5877 Segmentation fault in libjvm.so!VM_Exit::wait_for_threads_in_native_to_block() [breakpad?]
- Resolved
-
IMPALA-5879 Segmentation fault in llvm::Instruction::Instruction(llvm::Type*, unsigned int, llvm::Use*, unsigned int, llvm::Instruction*) [breakpad?]
- Resolved
-
IMPALA-5880 Segmentation fault in impala::HdfsScanNodeBase::Prepare(impala::RuntimeState*)
- Resolved
-
IMPALA-5882 Segmentation fault in MemPool::FindChunk(long, bool)
- Resolved
-
IMPALA-5883 Segmentation fault in ClientCacheHelper::ReleaseClient(void**) [breakpad?]
- Resolved
- is related to
-
IMPALA-6263 Assert hit during service restart Mutex.cpp:130: apache::thrift::concurrency::Mutex::impl::~impl(): Assertion `ret == 0' failed
- Resolved
- relates to
-
IMPALA-5891 PeriodicCounterUpdater should not rely on static initialisation and destruction order
- Resolved