Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3290

Catch std::bad_alloc from LLVM optimisation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • Impala 2.5.0
    • None
    • Backend
    • None

    Description

      I have a core dump locally, the stack trace is like:

      #0  0x00007f79a50dacc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f79a50de218 in __GI_abort () at abort.c:118
#2  0x00007f79a59ee6dd in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f79a59ec746 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f79a59ec791 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f79a59ec9a8 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00000000023065c8 in (anonymous namespace)::handle_oom(void* (*)(void*), void*, bool, bool) [clone .constprop.53] ()
#7  0x0000000002326fd0 in tc_newarray ()
#8  0x00000000021bbb94 in llvm::Instruction::setMetadata(unsigned int, llvm::MDNode*) ()
#9  0x00000000021a38c0 in llvm::Instruction::clone() const ()
#10 0x0000000001f5e9c4 in (anonymous namespace)::PruningFunctionCloner::CloneBlock(llvm::BasicBlock const*, std::vector<llvm::BasicBlock const*, std::allocator<llvm::BasicBlock const*> >&) ()
#11 0x0000000001f5fd43 in llvm::CloneAndPruneFunctionInto(llvm::Function*, llvm::Function const*, llvm::ValueMap<llvm::Value const*, llvm::WeakVH, llvm::ValueMapConfig<llvm::Value const*> >&, bool, llvm::SmallVectorImpl<llvm::ReturnInst*>&, char const*, llvm::ClonedCodeInfo*, llvm::DataLayout const*, llvm::Instruction*) ()
#12 0x0000000001f647b6 in llvm::InlineFunction(llvm::CallSite, llvm::InlineFunctionInfo&, bool) ()
#13 0x0000000001987f1f in llvm::Inliner::runOnSCC(llvm::CallGraphSCC&) ()
#14 0x0000000001fb9d6e in (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) ()
#15 0x00000000021c8158 in llvm::MPPassManager::runOnModule(llvm::Module&) ()
#16 0x00000000021c8331 in llvm::PassManagerImpl::run(llvm::Module&) ()
#17 0x00000000015325f5 in impala::LlvmCodeGen::OptimizeModule (this=0x7e209200) at /home/huaisi/Impala/be/src/codegen/llvm-codegen.cc:747
#18 0x0000000001531c55 in impala::LlvmCodeGen::FinalizeModule (this=0x7e209200) at /home/huaisi/Impala/be/src/codegen/llvm-codegen.cc:673
#19 0x0000000001874587 in impala::PlanFragmentExecutor::OptimizeLlvmModule (this=0x798f868) at /home/huaisi/Impala/be/src/runtime/plan-fragment-executor.cc:284
#20 0x0000000001874f86 in impala::PlanFragmentExecutor::Open (this=0x798f868) at /home/huaisi/Impala/be/src/runtime/plan-fragment-executor.cc:326
#21 0x000000000144d79c in impala::FragmentMgr::FragmentExecState::Exec (this=0x798f600) at /home/huaisi/Impala/be/src/service/fragment-exec-state.cc:53
#22 0x0000000001444fa5 in impala::FragmentMgr::FragmentThread (this=0x76a0d00, fragment_instance_id=...) at /home/huaisi/Impala/be/src/service/fragment-mgr.cc:86
#23 0x0000000001448b02 in boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>::operator() (this=0x127efc000, p=0x76a0d00, a1=...) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:165
#24 0x00000000014488bf in boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list0> (this=0x127efc010, f=..., a=...)
    at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/bind/bind.hpp:313
#25 0x00000000014481e9 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >::operator() (this=0x127efc000)
    at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20
#26 0x0000000001447b52 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >, void>::invoke
    (function_obj_ptr=...) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:153
#27 0x000000000125dd10 in boost::function0<void>::operator() (this=0x7f791a9f3d60) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:767
#28 0x0000000001509e3b in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name="exec-plan-fragment-e348c82278866d37:f5eb00d2bb759789", category="fragment-mgr", functor=..., thread_started=0x7f791d28eaa0)
    at /home/huaisi/Impala/be/src/util/thread.cc:316
#29 0x000000000151156c in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x7e4147c0, 
    f=@0x7e4147b8: 0x1509b76 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457
#30 0x00000000015114af in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x7e4147b8) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20
#31 0x0000000001511472 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x7e414600) at /home/huaisi/Impala/toolchain/boost-1.57.0/include/boost/thread/detail/thread.hpp:116
#32 0x000000000192ba1a in thread_proxy ()
#33 0x00007f79a7d2e182 in start_thread (arg=0x7f791a9f4700) at pthread_create.c:312
#34 0x00007f79a519e47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

      IMPALA.ERROR:

      tcmalloc: large alloc 1610612736 bytes == 0x3876b2000 @  0x2326ef6 0x21bbb94
tcmalloc: large alloc 3221225472 bytes == 0x4febb2000 @  0x2326ef6 0x21bbb94
tcmalloc: large alloc 2147483648 bytes == 0x7f77df170000 @  0x2326ef6 0x21d9368
tcmalloc: large alloc 6442450944 bytes == (nil) @  0x2326ef6 0x21bbb94
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Stack dump:
0.	Running pass 'CallGraph Pass Manager' on module ''.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3259 Codegen is not cancellable and can use a lot of CPU and memory

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5102 Handle uncaught exceptions in Impalad

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2506 Account for memory usage when preparing fragment

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              HuaisiXu Huaisi Xu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: