Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Distributed Exec
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      While running concurrent TPC-DS queries at 1TB scale I caught a few of these crashes.
      https://github.com/HenryR/Impala/tree/krpc change a53d43ad7bd10587e3d19295c937125cbf244765

      Program terminated with signal 6, Aborted.
      #0  0x0000003eab832625 in raise () from /lib64/libc.so.6
      Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.7.x86_64 jdk1.8.0_121-1.8.0_121-fcs.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6_7.4.x86_64 zlib-1.2.3-29.el6.x86_64
      (gdb) bt
      #0  0x0000003eab832625 in raise () from /lib64/libc.so.6
      #1  0x0000003eab833e05 in abort () from /lib64/libc.so.6
      #2  0x0000000002e476e4 in google::DumpStackTraceAndExit() ()
      #3  0x0000000002e3e15d in google::LogMessage::Fail() ()
      #4  0x0000000002e3fa02 in google::LogMessage::SendToLog() ()
      #5  0x0000000002e3db37 in google::LogMessage::Flush() ()
      #6  0x0000000002e410fe in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x0000000001fdc034 in impala::Sorter::Run::ConvertOffsetsToPtrs (this=0x7f9c51723290, tuple=0x7f815a66947d) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:1028
      #8  0x0000000001fe6002 in impala::Sorter::Run::GetNext<true> (this=0x7f9c51723290, output_batch=0x7f965ed49180, eos=0x7f93391ff7bf)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:868
      #9  0x0000000001fda9be in impala::Sorter::Run::GetNextBatch (this=0x7f9c51723290, output_batch=0x7f996a59ad20) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:797
      #10 0x0000000001febfca in boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>::operator() (this=0x7f996a59ad08, p=0x7f9c51723290, a1=0x7f996a59ad20)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/mem_fn_template.hpp:165
      #11 0x0000000001febc98 in boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> >::operator()<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list1<impala::RowBatch**&> > (this=0x7f996a59ad18, f=..., a=...) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind.hpp:303
      #12 0x0000000001feb62f in boost::_bi::bind_t<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> > >::operator()<impala::RowBatch**> (this=0x7f996a59ad08, a1=@0x7f93391ff968) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind_template.hpp:32
      #13 0x0000000001feaf02 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> > >, impala::Status, impala::RowBatch**>::invoke (function_obj_ptr=..., a0=0x7f996a59ad20)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:132
      #14 0x00000000018d7f40 in boost::function1<impala::Status, impala::RowBatch**>::operator() (this=0x7f996a59ad00, a0=0x7f996a59ad20)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:767
      #15 0x00000000018d7b0a in impala::SortedRunMerger::SortedRunWrapper::Advance (this=0x7f996a59ad00, transfer_batch=0x0, eos=0x7f93391ffb0f)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:76
      #16 0x00000000018d7453 in impala::SortedRunMerger::AdvanceMinRow (this=0x6a3564c0, transfer_batch=0x7f92f806da80)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:187
      #17 0x00000000018d7091 in impala::SortedRunMerger::GetNext (this=0x6a3564c0, output_batch=0x7f92f806da80, eos=0x236d09f0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:173
      #18 0x0000000001fdfd4c in impala::Sorter::GetNext (this=0x7f9c2abe8f20, output_batch=0x7f92f806da80, eos=0x236d09f0) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:1445
      #19 0x0000000001d6e53c in impala::SortNode::GetNext (this=0x7f9bb5ac7de0, state=0x7f987f361000, row_batch=0x7f92f806da80, eos=0x236d09f0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/sort-node.cc:103
      #20 0x0000000001d8f414 in impala::AnalyticEvalNode::ProcessChildBatches (this=0x236d0700, state=0x7f987f361000) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/analytic-eval-node.cc:610
      #21 0x0000000001d91ffb in impala::AnalyticEvalNode::GetNext (this=0x236d0700, state=0x7f987f361000, row_batch=0x7f95a3c629a0, eos=0x2922a11c)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/analytic-eval-node.cc:792
      #22 0x0000000001d6b507 in impala::SelectNode::GetNext (this=0x2922a000, state=0x7f987f361000, row_batch=0x7f9339200370, eos=0x7f93392004af)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/select-node.cc:76
      #23 0x0000000001d7421b in impala::TopNNode::Open (this=0x7f9914466000, state=0x7f987f361000) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/topn-node.cc:153
      #24 0x00000000018f6951 in impala::FragmentInstanceState::Open (this=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/fragment-instance-state.cc:265
      #25 0x00000000018f4413 in impala::FragmentInstanceState::Exec (this=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/fragment-instance-state.cc:88
      #26 0x00000000018b8f10 in impala::QueryState::ExecFInstance (this=0x1f70c800, fis=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/query-state.cc:333
      #27 0x00000000018b7c12 in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7f9339200bc8) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/query-state.cc:307
      #28 0x00000000018b98ed in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:153
      #29 0x00000000016bf878 in boost::function0<void>::operator() (this=0x7f9339200bc0)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:767
      #30 0x0000000001b282bf in impala::Thread::SuperviseThread (name=..., category=..., functor=..., thread_started=0x7f9340c0bc20)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/util/thread.cc:322
      #31 0x0000000001b30c2e in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x7f9c5e90e3c0, f=@0x7f9c5e90e3b8, a=...)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind.hpp:457
      #32 0x0000000001b30b71 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x7f9c5e90e3b8)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind_template.hpp:20
      #33 0x0000000001b30b34 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x7f9c5e90e200)
          at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/thread/detail/thread.hpp:116
      #34 0x000000000204865a in thread_proxy ()
      #35 0x0000003eabc07aa1 in start_thread () from /lib64/libpthread.so.0
      #36 0x0000003eab8e893d in clone () from /lib64/libc.so.6

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                mmulder Matthew Mulder
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: