Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 1.1.1
-
None
-
None
Description
Under certain conditions, HdfsScanNode will create thousands of threads until the impalad process crashes.
Attached is a stack dump from a recent build, which was produced by running the new HS2 test cases in a loop. The most interesting thread stack is excerpted below - the scan node is creating new threads when its coordinator is being torn down. It's not clear, however, if this is where the looping happens.
#0 0x00007fd1ded66ca1 in __nptl_create_event () at events.c:27 #1 0x00007fd1ded68ad5 in create_thread (newthread=0x4b52c28, attr=0x0, start_routine=0x7fd1e112c110 <thread_proxy>, arg=0x4b52c00) at ../nptl/sysdeps/pthread/createthread.c:231 #2 __pthread_create_2_1 (newthread=0x4b52c28, attr=0x0, start_routine=0x7fd1e112c110 <thread_proxy>, arg=0x4b52c00) at pthread_create.c:561 #3 0x00007fd1e112bf31 in boost::thread::start_thread() () from /usr/lib/libboost_thread.so.1.42.0 #4 0x0000000000cf8853 in boost::thread::thread<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), std::basic_string<char>, std::basic_string<char>, boost::function<void()>, impala::Promise<long int>*>(void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::function<void()>, impala::Promise<long> *) (this=0x903e980, f=0xcf6b70 <impala::Thread::SuperviseThread(std::string const&, std::string const&, impala::Thread::ThreadFunctor, impala::Promise<long>*)>, a1=..., a2=..., a3=..., a4=0x7fd1c1a6bd20) at /usr/include/boost/thread/detail/thread.hpp:259 #5 0x0000000000cf68d5 in impala::Thread::StartThread (this=0x77cce80, functor=...) at /home/henry/src/cloudera/impala/be/src/util/thread.cc:224 #6 0x0000000000f19a7a in impala::Thread::Thread<void (impala::HdfsScanNode::*)(), impala::HdfsScanNode*> (this=0x77cce80, category=..., name=..., f=@0x7fd1c1a6c000, a1=@0x7fd1c1a6c028) at /home/henry/src/cloudera/impala/be/src/util/thread.h:68 #7 0x0000000000f15b92 in impala::HdfsScanNode::ThreadTokenAvailableCb (this=0x83c0c00, pool=0x7510de0) at /home/henry/src/cloudera/impala/be/src/exec/hdfs-scan-node.cc:667 #8 0x0000000000f245f2 in boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>::operator() (this=0x7510e28, p=0x83c0c00, a1=0x7510de0) at /usr/include/boost/bind/mem_fn_template.hpp:165 #9 0x0000000000f2382c in boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> >::operator()<boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list1<impala::ThreadResourceMgr::ResourcePool*&> > (this=0x7510e38, f=..., a=...) at /usr/include/boost/bind/bind.hpp:313 #10 0x0000000000f229bf in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> > >::operator()<impala::ThreadResourceMgr::ResourcePool*> (this=0x7510e28, a1=@0x7fd1c1a6c140) at /usr/include/boost/bind/bind_template.hpp:32 #11 0x0000000000f20f65 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> > >, void, impala::ThreadResourceMgr::ResourcePool*>::invoke (function_obj_ptr=..., a0=0x7510de0) at /usr/include/boost/function/function_template.hpp:153 #12 0x0000000000dc1d1d in boost::function1<void, impala::ThreadResourceMgr::ResourcePool*>::operator() (this=0x7510e20, a0=0x7510de0) at /usr/include/boost/function/function_template.hpp:1013 #13 0x0000000000dcd799 in impala::ThreadResourceMgr::UpdatePoolQuotas (this=0x4dbf700, new_pool=0x0) at /home/henry/src/cloudera/impala/be/src/runtime/thread-resource-mgr.cc:104 #14 0x0000000000dcd4e2 in impala::ThreadResourceMgr::UnregisterPool (this=0x4dbf700, pool=0x8e5fbc0) at /home/henry/src/cloudera/impala/be/src/runtime/thread-resource-mgr.cc:86 #15 0x0000000000dc102a in impala::PlanFragmentExecutor::Close (this=0x5bbb980) at /home/henry/src/cloudera/impala/be/src/runtime/plan-fragment-executor.cc:467 #16 0x0000000000dbc77d in impala::PlanFragmentExecutor::~PlanFragmentExecutor (this=0x5bbb980, __in_chrg=<value optimized out>) at /home/henry/src/cloudera/impala/be/src/runtime/plan-fragment-executor.cc:71 #17 0x0000000000d70273 in boost::checked_delete<impala::PlanFragmentExecutor> (x=0x5bbb980) at /usr/include/boost/checked_delete.hpp:34 #18 0x0000000000d6bf57 in boost::scoped_ptr<impala::PlanFragmentExecutor>::~scoped_ptr (this=0x588f620, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/scoped_ptr.hpp:80 #19 0x0000000000d5cff7 in impala::Coordinator::~Coordinator (this=0x588f400, __in_chrg=<value optimized out>) at /home/henry/src/cloudera/impala/be/src/runtime/coordinator.cc:230 #20 0x0000000000bcf867 in boost::checked_delete<impala::Coordinator> (x=0x588f400) at /usr/include/boost/checked_delete.hpp:34 #21 0x0000000000bc1b21 in boost::scoped_ptr<impala::Coordinator>::~scoped_ptr (this=0x73012d8, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/scoped_ptr.hpp:80 #22 0x0000000000bbab65 in impala::ImpalaServer::QueryExecState::~QueryExecState (this=0x7301200, __in_chrg=<value optimized out>) at /home/henry/src/cloudera/impala/be/src/service/query-exec-state.h:55 #23 0x0000000000bee22e in boost::checked_delete<impala::ImpalaServer::QueryExecState> (x=0x7301200) at /usr/include/boost/checked_delete.hpp:34 #24 0x0000000000c08ee8 in boost::detail::sp_counted_impl_p<impala::ImpalaServer::QueryExecState>::dispose (this=0x5181140) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78 #25 0x0000000000b92ccc in boost::detail::sp_counted_base::release (this=0x5181140) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145 #26 0x0000000000b92d45 in boost::detail::shared_count::~shared_count (this=0x7fd1c1a6c598, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217 #27 0x0000000000bbb5f4 in boost::shared_ptr<impala::ImpalaServer::QueryExecState>::~shared_ptr (this=0x7fd1c1a6c590, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169 #28 0x0000000000ba4c4a in impala::ImpalaServer::UnregisterQuery (this=0x5cb1c00, query_id=...) at /home/henry/src/cloudera/impala/be/src/service/impala-server.cc:1003 #29 0x0000000000ba5ba6 in impala::ImpalaServer::CloseSessionInternal (this=0x5cb1c00, session_id=..., ignore_if_absent=false) at /home/henry/src/cloudera/impala/be/src/service/impala-server.cc:1131 #30 0x0000000000c0bec4 in impala::ImpalaServer::CloseSession (this=0x5cb1c00, return_val=..., request=...) at /home/henry/src/cloudera/impala/be/src/service/impala-hs2-server.cc:310