Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1865

Crash: impala::AggregateFunctions::StringValGetValue

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.2
    • Impala 2.2
    • None
    • None

    Description

      There were 10 randomly chosen TPCH queries executing concurrently on the 10 node CDH5 cluster on tpch300gb_parquet DB. mem_limit was not set.

      3 Nodes crashed at about the same time. Each node produced a different stack trace. Here's a stack trace from one of the nodes:

      #0  0x00000038a10328e5 in raise () from /lib64/libc.so.6
      #0  0x00000038a10328e5 in raise () from /lib64/libc.so.6
      #1  0x00000038a10340c5 in abort () from /lib64/libc.so.6
      #2  0x00007f40db21b155 in os::abort(bool) () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so
      #3  0x00007f40db39a087 in VMError::report_and_die() () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so
      #4  0x00007f40db21fadf in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_45-cloudera/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00000038a1089c08 in memcpy () from /lib64/libc.so.6
      #7  0x00000000011d1bdf in impala::AggregateFunctions::StringValGetValue (ctx=0x81123100, src=...) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exprs/aggregate-functions.cc:78
      #8  0x00000000011d1c1d in impala::AggregateFunctions::StringValSerializeOrFinalize (ctx=0x81123100, src=...) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exprs/aggregate-functions.cc:84
      #9  0x0000000001595249 in impala::AggFnEvaluator::SerializeOrFinalize (this=0xd084f00, agg_fn_ctx=0x81123100, src=0xe6ff14c0, dst_slot_desc=0xcf8cee0, dst=0xe6ff14c0, fn=0x11d1bfa) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exprs/agg-fn-evaluator.cc:486
      #10 0x000000000152c6c2 in impala::AggFnEvaluator::Serialize (this=0xd084f00, agg_fn_ctx=0x81123100, tuple=0xe6ff14c0) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exprs/agg-fn-evaluator.h:250
      #11 0x000000000152c81e in impala::AggFnEvaluator::Serialize (evaluators=std::vector of length 1, capacity 1 = {...}, fn_ctxs=std::vector of length 1, capacity 1 = {...}, dst=0xe6ff14c0) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exprs/agg-fn-evaluator.h:286
      #12 0x00000000015257ad in impala::PartitionedAggregationNode::FinalizeTuple (this=0x20782f00, agg_fn_ctxs=std::vector of length 1, capacity 1 = {...}, tuple=0xe6ff14c0, pool=0x15d417c0) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:677
      #13 0x0000000001522f9d in impala::PartitionedAggregationNode::CleanupHashTbl (this=0x20782f00, ctxs=std::vector of length 1, capacity 1 = {...}, it=...) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:355
      #14 0x0000000001524b98 in impala::PartitionedAggregationNode::Partition::Close (this=0xff57770, finalize_rows=true) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:556
      #15 0x000000000152322a in impala::PartitionedAggregationNode::Close (this=0x20782f00, state=0x116c5c00) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:378
      #16 0x000000000143ed75 in impala::PlanFragmentExecutor::Close (this=0xb009d10) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/runtime/plan-fragment-executor.cc:570
      #17 0x00000000010966f6 in impala::FragmentMgr::FragmentExecState::Exec (this=0xb009b00) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-exec-state.cc:50
      #18 0x000000000108db1c in impala::FragmentMgr::FragmentExecThread (this=0x8f0e540, exec_state=0xb009b00) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-mgr.cc:70
      #19 0x00000000010920f0 in boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>::operator() (this=0x1abc90a0, p=0x8f0e540, a1=0xb009b00) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:165
      #20 0x0000000001091ea9 in boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list0> (this=0x1abc90b0, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:313
      #21 0x0000000001091791 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> > >::operator() (this=0x1abc90a0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #22 0x00000000010910df in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::FragmentMgr::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::FragmentMgr::FragmentExecState*> > >, void>::invoke (function_obj_ptr=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
      #23 0x0000000000f21725 in boost::function0<void>::operator() (this=0x7f3ef594bcc0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
      #24 0x000000000117a090 in impala::Thread::SuperviseThread (name="exec-plan-fragment", category="impala-server", functor=..., thread_started=0x7f402f2efb90) at /usr/src/debug/impala-2.2.0-cdh5.5.0-SNAPSHOT/be/src/util/thread.cc:311
      #25 0x000000000118322c in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::string &, const std::string &, impala::Thread::ThreadFunctor, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x6f4531c0, f=@0x6f4531b8, a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:457
      #26 0x0000000001183173 in boost::_bi::bind_t<void, void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x6f4531b8) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #27 0x0000000001183132 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x6f453000) at /opt/toolchain/boost-pic-1.55.0/include/boost/thread/detail/thread.hpp:117
      #28 0x00000000015e3793 in thread_proxy ()
      #29 0x00000038a1407851 in start_thread () from /lib64/libpthread.so.0
      #30 0x00000038a10e894d in clone () from /lib64/libc.so.6
      

      Additional information is located on e1220.halxg.cloudera.com in /data/4/IMPALA-1865 directory.

      Attachments

        Activity

          People

            mjacobs Matthew Jacobs
            tarasbob Taras Bobrovytsky
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: