[IMPALA-1083] Deadlock in scan node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.3.1
• Fix Version/s: Impala 1.4
• Assignee: Skye Wanderman-Milne
• Reporter: Alan Choi
• Created: 11/Jul/14 18:54
• Updated: 20/Dec/15 00:05
• Resolved: 11/Jul/14 18:55

Description

Here's the stack:

#0 0x0000003a9200e264 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003a92009508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003a920093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0000000000d40d51 in boost::mutex::lock (this=can't compute CFA for this frame
) at /usr/include/boost/thread/pthread/mutex.hpp:52
#4 0x0000000000d4f89f in boost::unique_lock<boost::mutex>::lock (this=can't compute CFA for this frame
) at /usr/include/boost/thread/locks.hpp:412
#5 0x0000000000d4f79b in boost::unique_lock<boost::mutex>::unique_lock (this=can't compute CFA for this frame
) at /usr/include/boost/thread/locks.hpp:290
#6 0x000000000122841e in impala::HdfsScanNode::ThreadTokenAvailableCb (this=can't compute CFA for this frame
) at /usr/src/debug/impala-1.3.1-cdh5.0.2/be/src/exec/hdfs-scan-node.cc:730
#7 0x000000000123c90e in boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>::operator()
(this=can't compute CFA for this frame
) at /usr/include/boost/bind/mem_fn_template.hpp:165
#8 0x000000000123b49c in boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> >::operator()<boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list1<impala::ThreadResourceMgr::ResourcePool*&> > (this=can't compute CFA for this frame
) at /usr/include/boost/bind/bind.hpp:313
#9 0x0000000001239fd6 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> > >::operator()<impala::ThreadResourceMgr::ResourcePool*> (this=can't compute CFA for this frame
) at /usr/include/boost/bind/bind_template.hpp:32
#10 0x0000000001237afd in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::HdfsScanNode, impala::ThreadResourceMgr::ResourcePool*>, boost::_bi::list2<boost::_bi::value<impala::HdfsScanNode*>, boost::arg<1> > >, void, impala::ThreadResourceMgr::ResourcePool*>::invoke (function_obj_ptr=
can't compute CFA for this frame
) at /usr/include/boost/function/function_template.hpp:153
#11 0x0000000000de74e2 in boost::function1<void, impala::ThreadResourceMgr::ResourcePool*>::operator() (this=
can't compute CFA for this frame
) at /usr/include/boost/function/function_template.hpp:1013
#12 0x00000000011e8d0a in impala::ThreadResourceMgr::ResourcePool::ReleaseThreadToken (this=can't compute CFA for this frame
) at /usr/src/debug/impala-1.3.1-cdh5.0.2/be/src/runtime/thread-resource-mgr.h:279
#13 0x0000000001228963 in impala::HdfsScanNode::ScannerThread (this=can't compute CFA for this frame
) at /usr/src/debug/impala-1.3.1-cdh5.0.2/be/src/exec/hdfs-scan-node.cc:778
#14 0x000000000123d371 in boost::_mfi::mf0<void, impala::HdfsScanNode>::operator() (this=can't compute CFA for this frame
) at /usr/include/boost/bind/mem_fn_template.hpp:49

It's already fixed by commit 28add6ca by Skye on May 15.

Once this deadlock is hit, the whole cluster would appear to be hang. Impala won't be able to process any query.