[IMPALA-316] Impala segfaults executing count(*) query in impala::DiskIoMgr::queue_size (this=0x4048c80, reader=0x6740ea0) at disk-io-mgr.cc:801

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 0.1
• Fix Version/s: Impala 1.0
• Assignee: Nong Li
• Reporter: Ishaan Joshi
• Created: 25/Apr/13 17:25
• Updated: 25/Apr/13 20:12
• Resolved: 25/Apr/13 20:12

Description

This setting was added to the hive safety valve for impala to override the default timeout, which was 20 seconds.

<property>
    <name>hive.metastore.client.socket.timeout</name>
    <value>0</value>
</property>

I ran count(*)s over a large table for 10 or so iterations, I can consistently see a backend crashing [not the coordinator]. This does not happen when I remove the safety valve.
Stack:

#0  0x000000387d832885 in raise () from /lib64/libc.so.6
#1  0x000000387d834065 in abort () from /lib64/libc.so.6
#2  0x00007f77f0732727 in os::abort(bool) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
#3  0x00007f77f0885cc8 in VMError::report_and_die() () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
#4  0x00007f77f07390e5 in JVM_handle_linux_signal () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
#5  0x00007f77f07353ee in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x0000000000d5589d in std::_List_const_iterator<impala::DiskIoMgr::BufferDescriptor*>::operator++ (this=0x7f77bda32a60) at /usr/include/c++/4.4.6/bits/stl_list.h:223
#8  0x0000000000d58bf5 in std::__distance<std::_List_const_iterator<impala::DiskIoMgr::BufferDescriptor*> > (__first=Cannot access memory at address 0x10
) at /usr/include/c++/4.4.6/bits/stl_iterator_base_funcs.h:79
#9  0x0000000000d56cc8 in std::distance<std::_List_const_iterator<impala::DiskIoMgr::BufferDescriptor*> > (__first=, __last=) at /usr/include/c++/4.4.6/bits/stl_iterator_base_funcs.h:114
#10 0x0000000000d54fa5 in std::list<impala::DiskIoMgr::BufferDescriptor*, std::allocator<impala::DiskIoMgr::BufferDescriptor*> >::size (this=0x6740fe8) at /usr/include/c++/4.4.6/bits/stl_list.h:805
#11 0x0000000000d4a814 in impala::DiskIoMgr::queue_size (this=0x4048c80, reader=0x6740ea0) at /usr/src/debug/impala-1.0-SNAPSHOT/be/src/runtime/disk-io-mgr.cc:801
#12 0x0000000000ec034f in boost::_mfi::cmf1<long, impala::DiskIoMgr, impala::DiskIoMgr::ReaderContext*>::call<impala::DiskIoMgr* const, impala::DiskIoMgr::ReaderContext*> (this=0x5b4f520, u=@0x5b4f530, b1=@0x7f77bda32b88)
    at /usr/include/boost/bind/mem_fn_template.hpp:222
#13 0x0000000000ebfada in boost::_mfi::cmf1<long, impala::DiskIoMgr, impala::DiskIoMgr::ReaderContext*>::operator()<impala::DiskIoMgr*> (this=0x5b4f520, u=@0x5b4f530, a1=0x6740ea0) at /usr/include/boost/bind/mem_fn_template.hpp:232
#14 0x0000000000ebecb2 in boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::ReaderContext*> >::operator()<long, boost::_mfi::cmf1<long, impala::DiskIoMgr, impala::DiskIoMgr::ReaderContext*>, boost::_bi::list0> (this=0x5b4f530, f=..., a=...) at /usr/include/boost/bind/bind.hpp:303
#15 0x0000000000ebd3e3 in boost::_bi::bind_t<long, boost::_mfi::cmf1<long, impala::DiskIoMgr, impala::DiskIoMgr::ReaderContext*>, boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::ReaderContext*> > >::operator() (this=0x5b4f520) at /usr/include/boost/bind/bind_template.hpp:20
#16 0x0000000000ebafb8 in boost::detail::function::function_obj_invoker0<boost::_bi::bind_t<long, boost::_mfi::cmf1<long, impala::DiskIoMgr, impala::DiskIoMgr::ReaderContext*>, boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::ReaderContext*> > >, long>::invoke (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:132
#17 0x0000000000c95a9d in boost::function0<long>::operator() (this=0x6024f30) at /usr/include/boost/function/function_template.hpp:1013
#18 0x0000000000c9419e in impala::RuntimeProfile::PeriodicCounterUpdateLoop () at /usr/src/debug/impala-1.0-SNAPSHOT/be/src/util/runtime-profile.cc:779
#19 0x0000000000c8207d in boost::detail::thread_data<void (*)()>::run (this=0x6a9b860) at /usr/include/boost/thread/detail/thread.hpp:61
#20 0x0000000000f37843 in thread_proxy ()
#21 0x000000387dc077f1 in start_thread () from /lib64/libpthread.so.0
#22 0x000000387d8e570d in clone () from /lib64/libc.so.6

I have a core available.