Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
Impala 1.0
-
None
-
None
Description
I've attached the stacks of the impalad that has hung. There's a bunch of threads in due due to IMPALA-313. The interesting thread is thread 5 (below) which is waiting for the build thread to finish. I don't see the build thread stack anywhere in the process so presumably it has finished or something bizarre has happened. I can't repro this very often at all (and as far as I can tell jenkins master has never hit it).
Thread 5 (Thread 0x7fa2b0d9a700 (LWP 9006)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x0000000000ca4728 in boost::condition_variable::wait (this=0x23cfdcd0, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:20
#2 0x0000000000e3971f in boost::detail::future_object_base::wait (this=0x23cfdc80, rethrow=true) at /usr/include/boost/thread/future.hpp:161
#3 0x0000000000e4031f in boost::detail::future_object<impala::Status>::get (this=0x23cfdc80) at /usr/include/boost/thread/future.hpp:325
#4 0x0000000000e3c8da in boost::unique_future<impala::Status>::get (this=0x7fa2b0d98d50) at /usr/include/boost/thread/future.hpp:679
#5 0x0000000000e33702 in impala::HashJoinNode::Open (this=0x65b2780, state=0x8268d80) at /home/nong/Projects/CDHImpala/be/src/exec/hash-join-node.cc:237
#6 0x0000000000e336d0 in impala::HashJoinNode::Open (this=0x7e90500, state=0x8268d80) at /home/nong/Projects/CDHImpala/be/src/exec/hash-join-node.cc:232
#7 0x0000000000eadcfe in impala::AggregationNode::Open (this=0x8ffe200, state=0x8268d80) at /home/nong/Projects/CDHImpala/be/src/exec/aggregation-node.cc:173
#8 0x0000000000cf1875 in impala::PlanFragmentExecutor::OpenInternal (this=0x4c53738) at /home/nong/Projects/CDHImpala/be/src/runtime/plan-fragment-executor.cc:241
#9 0x0000000000cf1698 in impala::PlanFragmentExecutor::Open (this=0x4c53738) at /home/nong/Projects/CDHImpala/be/src/runtime/plan-fragment-executor.cc:227
#10 0x0000000000b050fb in impala::ImpalaServer::FragmentExecState::Exec (this=0x4c53700) at /home/nong/Projects/CDHImpala/be/src/service/impala-server.cc:487
#11 0x0000000000b18f9d in impala::ImpalaServer::RunExecPlanFragment (this=0x43b5c00, exec_state=0x4c53700) at /home/nong/Projects/CDHImpala/be/src/service/impala-server.cc:1736
#12 0x0000000000b81ab2 in boost::_mfi::mf1<void, impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>::operator() (this=0xbc9e550, p=0x43b5c00, a1=0x4c53700) at /usr/include/boost/bind/mem_fn_template.hpp:165
#13 0x0000000000b81187 in boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>, boost::_bi::value<impala::ImpalaServer::FragmentExecState*> >::operator()<boost::_mfi::mf1<void, impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>, boost::_bi::list0> (this=0xbc9e560, f=..., a=...) at /usr/include/boost/bind/bind.hpp:313
#14 0x0000000000b80140 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>, boost::_bi::value<impala::ImpalaServer::FragmentExecState*> > >::operator() (this=0xbc9e550) at /usr/include/boost/bind/bind_template.hpp:20
#15 0x0000000000b7efb8 in boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::ImpalaServer, impala::ImpalaServer::FragmentExecState*>, boost::_bi::list2<boost::_bi::value<impala::ImpalaServer*>, boost::_bi::value<impala::ImpalaServer::FragmentExecState*> > > >::run (this=0xbc9e420) at /usr/include/boost/thread/detail/thread.hpp:56
#16 0x00007fa306f1a16e in thread_proxy () from /usr/lib/libboost_thread.so.1.42.0
#17 0x00007fa304b55d8c in start_thread (arg=0x7fa2b0d9a700) at pthread_create.c:304
#18 0x00007fa30410004d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()