[IMPALA-6262] Crash Impalad [ DataSink::Create fail which cause profile nullptr ]

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0, Impala 2.11.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Epic Color: ghx-label-9
• Assignee: Michael Ho
• Reporter: yyzzjj
• Created: 30/Nov/17 12:54
• Updated: 07/Dec/17 17:52
• Resolved: 07/Dec/17 17:52

Description

query got error status : Invalid plan. TupleIsNullPredicate has invalid tuple idx
then impalad crash coredump show the next stack:

I1130 21:09:16.887727  7744 status.cc:125] Invalid plan. TupleIsNullPredicate has invalid tuple idx.
    @          0x165a243  impala::Status::Status()
    @          0x2a02833  impala::TupleIsNullPredicate::Init()
    @          0x29c087b  impala::ScalarExpr::Init()
    @          0x29c087b  impala::ScalarExpr::Init()
    @          0x2a03046  impala::ScalarFnCall::Init()
    @          0x29c087b  impala::ScalarExpr::Init()
    @          0x29bef56  impala::ScalarExpr::Create()
    @          0x29bf0f8  impala::ScalarExpr::Create()
    @          0x29bf32a  impala::ScalarExpr::Create()
    @          0x1b636bf  impala::DataSink::Init()
    @          0x1b63491  impala::DataSink::Create()
    @          0x18a9556  impala::FragmentInstanceState::Prepare()
    @          0x18a80f7  impala::FragmentInstanceState::Exec()
    @          0x189193c  impala::QueryState::ExecFInstance()
    @          0x18901fe  _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
    @          0x189257b  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x17deeb6  boost::function0<>::operator()()
    @          0x1ad03a1  impala::Thread::SuperviseThread()
    @          0x1ad8f56  boost::_bi::list4<>::operator()<>()
    @          0x1ad8e99  boost::_bi::bind_t<>::operator()()
    @          0x1ad8e5c  boost::detail::thread_data<>::run()
    @          0x2d875ea  thread_proxy
    @       0x3fbc6079d1  (unknown)
    @       0x3fbbee886Wrote minidump to /export/servers/impala/logs/minidumps/impalad/2ec94d40-a178-4d15-c880a592-8a2b52a1.dmp



(gdb) bt
#0  0x000000359e032925 in raise () from /lib64/libc.so.6
#1  0x000000359e034105 in abort () from /lib64/libc.so.6
#2  0x00007fce2a0a0a55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera///jre/lib/amd64/server/libjvm.so
#3  0x00007fce2a220f87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera///jre/lib/amd64/server/libjvm.so
#4  0x00007fce2a0a596f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera///jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x0000000001810f36 in std::_Rb_tree<std::string, std::pair<std::string const, impala::RuntimeProfile::Counter*>, std::_Select1st<std::pair<std::string const, impala::RuntimeProfile::Counter*> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::_M_begin (this=0x20) at /usr/local/gcc-4.9.3/include/c++/4.9.3/bits/stl_tree.h:518
#7  0x000000000180f88e in std::_Rb_tree<std::string, std::pair<std::string const, impala::RuntimeProfile::Counter*>, std::_Select1st<std::pair<std::string const, impala::RuntimeProfile::Counter*> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::lower_bound (this=0x20, __k=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/bits/stl_tree.h:927
#8  0x000000000180e5a3 in std::map<std::string, impala::RuntimeProfile::Counter*, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::lower_bound (
    this=0x20, __x=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/bits/stl_map.h:902
#9  0x000000000180d290 in std::map<std::string, impala::RuntimeProfile::Counter*, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::operator[] (
    this=0x20, __k=...) at /usr/local/gcc-4.9.3/include/c++/4.9.3/bits/stl_map.h:496
#10 0x000000000183a396 in impala::RuntimeProfile::total_time_counter (this=0x0) at /export/ldb/online/kudu_rpc_branch/be/src/util/runtime-profile.h:257
#11 0x000000000290e31a in impala::KuduTableSink::Close (this=0x107aa300, state=0xe3fba80) at /export/ldb/online/kudu_rpc_branch/be/src/exec/kudu-table-sink.cc:354
#12 0x00000000018ab06e in impala::FragmentInstanceState::Close (this=0xe879680) at /export/ldb/online/kudu_rpc_branch/be/src/runtime/fragment-instance-state.cc:290
#13 0x00000000018a8418 in impala::FragmentInstanceState::Exec (this=0xe879680) at /export/ldb/online/kudu_rpc_branch/be/src/runtime/fragment-instance-state.cc:95
#14 0x000000000189193c in impala::QueryState::ExecFInstance (this=0xd4d6d00, fis=0xe879680) at /export/ldb/online/kudu_rpc_branch/be/src/runtime/query-state.cc:385
#15 0x00000000018901fe in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7fcd5be4cbc8) at /export/ldb/online/kudu_rpc_branch/be/src/runtime/query-state.cc:328
#16 0x000000000189257b in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (
    function_obj_ptr=...) at /export/ldb/online/kudu_rpc_branch/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#17 0x00000000017deeb6 in boost::function0<void>::operator() (this=0x7fcd5be4cbc0) at /export/ldb/online/kudu_rpc_branch/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#18 0x0000000001ad03a1 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., 
    thread_started=0x7fcd62c56ab0) at /export/ldb/online/kudu_rpc_branch/be/src/util/thread.cc:352
#19 0x0000000001ad8f56 in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0xcf703c0, 
    f=@0xcf703b8: 0x1ad0082 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
    at /export/ldb/online/kudu_rpc_branch/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#20 0x0000000001ad8e99 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0xcf703b8)
    at /export/ldb/online/kudu_rpc_branch/toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#21 0x0000000001ad8e5c in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0xcf70200)
    at /export/ldb/online/kudu_rpc_branch/toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#22 0x0000000002d875ea in thread_proxy ()
#23 0x000000359e4079d1 in start_thread () from /lib64/libpthread.so.0
#24 0x000000359e0e8b6d in clone () from /lib64/libc.so.6

Issue Links

• is broken by: IMPALA-4080 Share codegen work between fragment instances
• relates to: IMPALA-6280 Invalid plan for sorted INSERT with an outer join and null checking
• relates to: IMPALA-5403 Use RETURN_IF_ERROR to induce runtime failure scenarios for testing purposes