[IMPALA-6474] Impala crash caused by running TPCH on Kudu for many hours

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.12.0
• Component/s: Backend
• Labels: crash
• Epic Color: ghx-label-4
• Assignee: Tianyi Wang
• Reporter: Taras Bobrovytsky
• Created: 03/Feb/18 00:02
• Updated: 18/May/18 19:36
• Resolved: 18/May/18 19:36

Description

I ran TPCH on Kudu in a loop overnight on my machine on an ASAN build. It crashed after a few hours.

Here's the stack trace from hs_err_pid file:

Stack: [0x00007f2e10b8f000,0x00007f2e11390000], sp=0x00007f2e1138c0a0, free space=8180k 
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) 
C [impalad+0x16ccd40] impala::RuntimeProfile::EventSequence::AddNewerEvents(std::vector<long, std::allocator<long> > const&, std::vector<std::string, std::allocator<std::string> > const&)+0x1a0
C [impalad+0x16be743] impala::RuntimeProfile::Update(std::vector<impala::TRuntimeProfileNode, std::allocator<impala::TRuntimeProfileNode> > const&, int*)+0xd53
C [impalad+0x16bd85f] impala::RuntimeProfile::Update(impala::TRuntimeProfileTree const&)+0xdf
C [impalad+0x2a9d69d] impala::Coordinator::BackendState::InstanceStats::Update(impala::TFragmentInstanceExecStatus const&, impala::Coordinator::ExecSummary*, impala:: ProgressUpdater*)+0x18d
C [impalad+0x2a9cba0] impala::Coordinator::BackendState::ApplyExecStatusReport(impala::TReportExecStatusParams const&, impala::Coordinator::ExecSummary*, impala::ProgressUpdater*)+0x4c0
C [impalad+0x2a82cef] impala::Coordinator::UpdateBackendExecStatus(impala::TReportExecStatusParams const&)+0x2cf
C [impalad+0x14a8df6] impala::ImpalaServer::ReportExecStatus(impala::TReportExecStatusResult&, impala::TReportExecStatusParams const&)+0x216
C [impalad+0x1598368] impala::ImpalaInternalService::ReportExecStatus(impala::TReportExecStatusResult&, impala::TReportExecStatusParams const&)+0x118
C [impalad+0x27a7d54] impala::ImpalaInternalServiceProcessor::process_ReportExecStatus(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)+0x3c4
C [impalad+0x27a6dfc] impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::string const&, int, void*)+0x3ac
C [impalad+0xfe4614] apache::thrift::TDispatchProcessor::process(boost::shared_ptr<apache::thrift::protocol::TProtocol>, boost::shared_ptr<apache::thrift::protocol::TProtocol>, void*)+0x174
C [impalad+0x12acee5] apache::thrift::server::TAcceptQueueServer::Task::run()+0x425 
C [impalad+0x12a5dda] impala::ThriftThread::RunRunnable(boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*)+0xea
C [impalad+0x12a8541] boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator()(impala:: ThriftThread*, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*) const+0x121
C [impalad+0x12a838d] void boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi:: value<impala::Promise<unsigned long>*> >::operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>&, boost::_bi::list0&, int)+0x10d
C [impalad+0x12a8232] boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala:: Promise<unsigned long>*> > >::operator()()+0xa2
C [impalad+0x12be473] boost::function0<void>::operator()() const+0xb3 
C [impalad+0x17174d4] impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()()>, impala::Promise<long>*)+0x414
C [impalad+0x17223a6] void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()()> >, boost::_bi:: value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi:: type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()()>, impala::Promise<long>*), boost::_bi::list0&, int)+0x136
C [impalad+0x1722222] boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi:: value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()()+0xa2

No core dumps were generated because this was as ASAN build. It doesn't look like this has anything to do with with ASAN though.