Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1841

Stress job causes Impala crash (impala::ExecNode::Close) (impala::BufferedBlockMgr::DeleteBlock)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.2
    • Fix Version/s: Impala 2.2
    • Component/s: None
    • Labels:
      None

      Description

      Impala crashes on the CDH5 10 node cluster when 10 queries are executing concurrently.

      In this run: http://sandbox.jenkins.cloudera.com/view/Impala/view/experimental/job/Random-Stress/18/console
      a node crashes even though no queries were sent to it.
      Stack Trace:

      #0  0x00000030770328e5 in raise () from /lib64/libc.so.6
      #0  0x00000030770328e5 in raise () from /lib64/libc.so.6
      #1  0x00000030770340c5 in abort () from /lib64/libc.so.6
      #2  0x0000000001f09da9 in google::DumpStackTraceAndExit () at src/utilities.cc:147
      #3  0x0000000001f013cd in google::LogMessage::Fail () at src/logging.cc:1315
      #4  0x0000000001f04f67 in google::LogMessage::SendToLog (this=0x7fa8d3307c30) at src/logging.cc:1269
      #5  0x0000000001f0399a in google::LogMessage::Flush (this=0x7fa8d3307c30) at src/logging.cc:1138
      #6  0x0000000001f03c6d in google::LogMessageFatal::~LogMessageFatal (this=0x7fa8d3307c30, __in_chrg=<value optimized out>) at src/logging.cc:1836
      #7  0x0000000000f52854 in impala::BufferedBlockMgr::DeleteBlock (this=0x44a2d440, block=0x8fc38c0) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:784
      #8  0x0000000000f4bae9 in impala::BufferedBlockMgr::Block::Delete (this=0x8fc38c0) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:135
      #9  0x00000000015830ea in impala::HashTable::Close (this=0xc3d9c20) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/hash-table.cc:249
      #10 0x000000000154083b in impala::PartitionedHashJoinNode::Partition::BuildHashTableInternal<false> (this=0xf2fc930, state=0xb165800, built=0x7fa8d330805d) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:363
      #11 0x000000000153271c in impala::PartitionedHashJoinNode::Partition::BuildHashTable (this=0xf2fc930, state=0xb165800, built=0x7fa8d330805d, add_probe_filters=false) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:290
      #12 0x00000000015395a0 in impala::PartitionedHashJoinNode::BuildHashTables (this=0xd822c00, state=0xb165800) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:1092
      #13 0x0000000001534b1f in impala::PartitionedHashJoinNode::ProcessBuildInput (this=0xd822c00, state=0xb165800, level=0) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:581
      #14 0x00000000015337ac in impala::PartitionedHashJoinNode::ConstructBuildSide (this=0xd822c00, state=0xb165800) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:486
      #15 0x0000000001571035 in impala::BlockingJoinNode::BuildSideThread (this=0xd822c00, state=0xb165800, status=0x7fa926591aa0) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/exec/blocking-join-node.cc:133
      #16 0x0000000001572be8 in boost::_mfi::mf2<void, impala::BlockingJoinNode, impala::RuntimeState*, impala::Promise<impala::Status>*>::operator() (this=0xfa885a0, p=0xd822c00, a1=0xb165800, a2=0x7fa926591aa0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:280
      #17 0x0000000001572a94 in boost::_bi::list3<boost::_bi::value<impala::BlockingJoinNode*>, boost::_bi::value<impala::RuntimeState*>, boost::_bi::value<impala::Promise<impala::Status>*> >::operator()<boost::_mfi::mf2<void, impala::BlockingJoinNode, impala::RuntimeState*, impala::Promise<impala::Status>*>, boost::_bi::list0> (this=0xfa885b0, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:392
      #18 0x0000000001572833 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::BlockingJoinNode, impala::RuntimeState*, impala::Promise<impala::Status>*>, boost::_bi::list3<boost::_bi::value<impala::BlockingJoinNode*>, boost::_bi::value<impala::RuntimeState*>, boost::_bi::value<impala::Promise<impala::Status>*> > >::operator() (this=0xfa885a0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #19 0x0000000001572767 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::BlockingJoinNode, impala::RuntimeState*, impala::Promise<impala::Status>*>, boost::_bi::list3<boost::_bi::value<impala::BlockingJoinNode*>, boost::_bi::value<impala::RuntimeState*>, boost::_bi::value<impala::Promise<impala::Status>*> > >, void>::invoke (function_obj_ptr=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
      #20 0x0000000000f1f159 in boost::function0<void>::operator() (this=0x7fa8d3308cc0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
      #21 0x0000000001176fb0 in impala::Thread::SuperviseThread (name="build thread", category="PartitionedHashJoinNode", functor=..., thread_started=0x7fa9265918a0) at /usr/src/debug/impala-2.1.0-cdh5.4.0-SNAPSHOT/be/src/util/thread.cc:311
      #22 0x000000000118014c in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::string &, const std::string &, impala::Thread::ThreadFunctor, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x34ddc3c0, f=@0x34ddc3b8, a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:457
      #23 0x0000000001180093 in boost::_bi::bind_t<void, void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x34ddc3b8) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #24 0x0000000001180052 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::string&, const std::string&, impala::Thread::ThreadFunctor, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x34ddc200) at /opt/toolchain/boost-pic-1.55.0/include/boost/thread/detail/thread.hpp:117
      #25 0x00000000015df733 in thread_proxy ()
      #26 0x0000003077407851 in start_thread () from /lib64/libpthread.so.0
      #27 0x00000030770e894d in clone () from /lib64/libc.so.6
      

        Attachments

          Activity

            People

            • Assignee:
              ippokratis Ippokratis Pandis
              Reporter:
              tarasbob Taras Bobrovytsky
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: