[IMPALA-1851] Crash: Race condition trying to destruct DataStreamRecvr triggered by TransmitData()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.2
• Assignee: casey
• Reporter: casey
• Created: 06/Mar/15 02:03
• Updated: 13/Mar/15 22:32
• Resolved: 13/Mar/15 22:32

Description

After fixing IMPALA-1835, there is a new crash. It's likely related to the same cancellation race.

#0  0x00007f80a2fade37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f80a2faf528 in __GI_abort () at abort.c:89
#2  0x00007f80a218fcf5 in os::abort(bool) ()
   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
#3  0x00007f80a23114f7 in VMError::report_and_die() ()
   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
#4  0x00007f80a2194c0f in JVM_handle_linux_signal ()
   from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00007f80a38c8e8a in std::__detail::_List_node_base::_M_unhook() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000000000963679 in _M_erase (this=<optimized out>, __position=...)
    at /usr/include/c++/4.9/bits/stl_list.h:1697
Python Exception <class 'ValueError'> Cannot find type std::list<impala::MemTracker*, std::allocator<impala::MemTracker*> >::iterator::_Node: 
#8  erase (this=<optimized out>, __position=) at /usr/include/c++/4.9/bits/list.tcc:154
#9  impala::MemTracker::UnregisterFromParent (this=0x799d4a0)
    at /home/casey/Impala/be/src/runtime/mem-tracker.cc:129
#10 0x0000000000bddf86 in impala::DataStreamRecvr::~DataStreamRecvr (this=0x7987400, 
    __in_chrg=<optimized out>)
    at /home/casey/Impala/be/src/runtime/data-stream-recvr.cc:358
#11 0x0000000000bdd422 in checked_delete<impala::DataStreamRecvr> (x=0x7987400)
    at /usr/include/boost/checked_delete.hpp:34
#12 boost::detail::sp_counted_impl_p<impala::DataStreamRecvr>::dispose (
    this=<optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#13 0x0000000000768879 in release (this=0x8c78d40)
    at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#14 boost::detail::shared_count::~shared_count (this=<optimized out>, 
    __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:371
#15 0x0000000000bdc0c9 in ~shared_ptr (this=0x7f801fe9f620, __in_chrg=<optimized out>)
    at /usr/include/boost/smart_ptr/shared_ptr.hpp:328
#16 impala::DataStreamMgr::AddData (this=0x72ed170, fragment_instance_id=..., 
    dest_node_id=7, thrift_batch=..., sender_id=1)
    at /home/casey/Impala/be/src/runtime/data-stream-mgr.cc:104
#17 0x00000000009b4f47 in impala::ImpalaServer::TransmitData (this=0x6cc1b80, 
    return_val=..., params=...) at /home/casey/Impala/be/src/service/impala-server.cc:969
#18 0x0000000000b3434a in impala::ImpalaInternalServiceProcessor::process_TransmitData (
    this=0x7d66540, seqid=0, iprot=<optimized out>, oprot=0xd308a00, 
    callContext=<optimized out>)
    at /home/casey/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1111
#19 0x0000000000b32b89 in impala::ImpalaInternalServiceProcessor::dispatchCall (
    this=0x7d66540, iprot=0x82bf280, oprot=0xd308a00, fname="TransmitData", seqid=0, 
    callContext=0xd308e80)
    at /home/casey/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:922