Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5131 Known issues with KRPC (IMPALA-2567)
  3. IMPALA-5554

crash in impala::Sorter::Run::ConvertOffsetsToPtrs

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0
    • Impala 2.10.0
    • Distributed Exec
    • None
    • ghx-label-7

    Description

      While running concurrent TPC-DS queries at 1TB scale I caught a few of these crashes.
      https://github.com/HenryR/Impala/tree/krpc change a53d43ad7bd10587e3d19295c937125cbf244765

      Program terminated with signal 6, Aborted.
#0  0x0000003eab832625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.7.x86_64 jdk1.8.0_121-1.8.0_121-fcs.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6_7.4.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x0000003eab832625 in raise () from /lib64/libc.so.6
#1  0x0000003eab833e05 in abort () from /lib64/libc.so.6
#2  0x0000000002e476e4 in google::DumpStackTraceAndExit() ()
#3  0x0000000002e3e15d in google::LogMessage::Fail() ()
#4  0x0000000002e3fa02 in google::LogMessage::SendToLog() ()
#5  0x0000000002e3db37 in google::LogMessage::Flush() ()
#6  0x0000000002e410fe in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x0000000001fdc034 in impala::Sorter::Run::ConvertOffsetsToPtrs (this=0x7f9c51723290, tuple=0x7f815a66947d) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:1028
#8  0x0000000001fe6002 in impala::Sorter::Run::GetNext<true> (this=0x7f9c51723290, output_batch=0x7f965ed49180, eos=0x7f93391ff7bf)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:868
#9  0x0000000001fda9be in impala::Sorter::Run::GetNextBatch (this=0x7f9c51723290, output_batch=0x7f996a59ad20) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:797
#10 0x0000000001febfca in boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>::operator() (this=0x7f996a59ad08, p=0x7f9c51723290, a1=0x7f996a59ad20)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/mem_fn_template.hpp:165
#11 0x0000000001febc98 in boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> >::operator()<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list1<impala::RowBatch**&> > (this=0x7f996a59ad18, f=..., a=...) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind.hpp:303
#12 0x0000000001feb62f in boost::_bi::bind_t<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> > >::operator()<impala::RowBatch**> (this=0x7f996a59ad08, a1=@0x7f93391ff968) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind_template.hpp:32
#13 0x0000000001feaf02 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<impala::Status, boost::_mfi::mf1<impala::Status, impala::Sorter::Run, impala::RowBatch**>, boost::_bi::list2<boost::_bi::value<impala::Sorter::Run*>, boost::arg<1> > >, impala::Status, impala::RowBatch**>::invoke (function_obj_ptr=..., a0=0x7f996a59ad20)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:132
#14 0x00000000018d7f40 in boost::function1<impala::Status, impala::RowBatch**>::operator() (this=0x7f996a59ad00, a0=0x7f996a59ad20)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:767
#15 0x00000000018d7b0a in impala::SortedRunMerger::SortedRunWrapper::Advance (this=0x7f996a59ad00, transfer_batch=0x0, eos=0x7f93391ffb0f)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:76
#16 0x00000000018d7453 in impala::SortedRunMerger::AdvanceMinRow (this=0x6a3564c0, transfer_batch=0x7f92f806da80)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:187
#17 0x00000000018d7091 in impala::SortedRunMerger::GetNext (this=0x6a3564c0, output_batch=0x7f92f806da80, eos=0x236d09f0)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorted-run-merger.cc:173
#18 0x0000000001fdfd4c in impala::Sorter::GetNext (this=0x7f9c2abe8f20, output_batch=0x7f92f806da80, eos=0x236d09f0) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/sorter.cc:1445
#19 0x0000000001d6e53c in impala::SortNode::GetNext (this=0x7f9bb5ac7de0, state=0x7f987f361000, row_batch=0x7f92f806da80, eos=0x236d09f0)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/sort-node.cc:103
#20 0x0000000001d8f414 in impala::AnalyticEvalNode::ProcessChildBatches (this=0x236d0700, state=0x7f987f361000) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/analytic-eval-node.cc:610
#21 0x0000000001d91ffb in impala::AnalyticEvalNode::GetNext (this=0x236d0700, state=0x7f987f361000, row_batch=0x7f95a3c629a0, eos=0x2922a11c)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/analytic-eval-node.cc:792
#22 0x0000000001d6b507 in impala::SelectNode::GetNext (this=0x2922a000, state=0x7f987f361000, row_batch=0x7f9339200370, eos=0x7f93392004af)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/select-node.cc:76
#23 0x0000000001d7421b in impala::TopNNode::Open (this=0x7f9914466000, state=0x7f987f361000) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/exec/topn-node.cc:153
#24 0x00000000018f6951 in impala::FragmentInstanceState::Open (this=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/fragment-instance-state.cc:265
#25 0x00000000018f4413 in impala::FragmentInstanceState::Exec (this=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/fragment-instance-state.cc:88
#26 0x00000000018b8f10 in impala::QueryState::ExecFInstance (this=0x1f70c800, fis=0x7f97963c4280) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/query-state.cc:333
#27 0x00000000018b7c12 in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7f9339200bc8) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/query-state.cc:307
#28 0x00000000018b98ed in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:153
#29 0x00000000016bf878 in boost::function0<void>::operator() (this=0x7f9339200bc0)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:767
#30 0x0000000001b282bf in impala::Thread::SuperviseThread (name=..., category=..., functor=..., thread_started=0x7f9340c0bc20)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/util/thread.cc:322
#31 0x0000000001b30c2e in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x7f9c5e90e3c0, f=@0x7f9c5e90e3b8, a=...)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind.hpp:457
#32 0x0000000001b30b71 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x7f9c5e90e3b8)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind_template.hpp:20
#33 0x0000000001b30b34 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x7f9c5e90e200)
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p2/include/boost/thread/detail/thread.hpp:116
#34 0x000000000204865a in thread_proxy ()
#35 0x0000003eabc07aa1 in start_thread () from /lib64/libpthread.so.0
#36 0x0000003eab8e893d in clone () from /lib64/libc.so.6

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3317 Stress test failure: sorter.cc:745] Check failed: i == 0 (1 vs. 0)

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              tarmstrong Tim Armstrong
              mmulder Matthew Mulder
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: