Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1851

Crash: Race condition trying to destruct DataStreamRecvr triggered by TransmitData()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.2
    • Fix Version/s: Impala 2.2
    • Component/s: None
    • Labels:
      None

      Description

      After fixing IMPALA-1835, there is a new crash. It's likely related to the same cancellation race.

      #0  0x00007f80a2fade37 in __GI_raise (sig=sig@entry=6)
          at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
      #1  0x00007f80a2faf528 in __GI_abort () at abort.c:89
      #2  0x00007f80a218fcf5 in os::abort(bool) ()
         from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
      #3  0x00007f80a23114f7 in VMError::report_and_die() ()
         from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
      #4  0x00007f80a2194c0f in JVM_handle_linux_signal ()
         from /usr/lib/jvm/java-7-oracle/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00007f80a38c8e8a in std::__detail::_List_node_base::_M_unhook() ()
         from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #7  0x0000000000963679 in _M_erase (this=<optimized out>, __position=...)
          at /usr/include/c++/4.9/bits/stl_list.h:1697
      Python Exception <class 'ValueError'> Cannot find type std::list<impala::MemTracker*, std::allocator<impala::MemTracker*> >::iterator::_Node: 
      #8  erase (this=<optimized out>, __position=) at /usr/include/c++/4.9/bits/list.tcc:154
      #9  impala::MemTracker::UnregisterFromParent (this=0x799d4a0)
          at /home/casey/Impala/be/src/runtime/mem-tracker.cc:129
      #10 0x0000000000bddf86 in impala::DataStreamRecvr::~DataStreamRecvr (this=0x7987400, 
          __in_chrg=<optimized out>)
          at /home/casey/Impala/be/src/runtime/data-stream-recvr.cc:358
      #11 0x0000000000bdd422 in checked_delete<impala::DataStreamRecvr> (x=0x7987400)
          at /usr/include/boost/checked_delete.hpp:34
      #12 boost::detail::sp_counted_impl_p<impala::DataStreamRecvr>::dispose (
          this=<optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
      #13 0x0000000000768879 in release (this=0x8c78d40)
          at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
      #14 boost::detail::shared_count::~shared_count (this=<optimized out>, 
          __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:371
      #15 0x0000000000bdc0c9 in ~shared_ptr (this=0x7f801fe9f620, __in_chrg=<optimized out>)
          at /usr/include/boost/smart_ptr/shared_ptr.hpp:328
      #16 impala::DataStreamMgr::AddData (this=0x72ed170, fragment_instance_id=..., 
          dest_node_id=7, thrift_batch=..., sender_id=1)
          at /home/casey/Impala/be/src/runtime/data-stream-mgr.cc:104
      #17 0x00000000009b4f47 in impala::ImpalaServer::TransmitData (this=0x6cc1b80, 
          return_val=..., params=...) at /home/casey/Impala/be/src/service/impala-server.cc:969
      #18 0x0000000000b3434a in impala::ImpalaInternalServiceProcessor::process_TransmitData (
          this=0x7d66540, seqid=0, iprot=<optimized out>, oprot=0xd308a00, 
          callContext=<optimized out>)
          at /home/casey/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1111
      #19 0x0000000000b32b89 in impala::ImpalaInternalServiceProcessor::dispatchCall (
          this=0x7d66540, iprot=0x82bf280, oprot=0xd308a00, fname="TransmitData", seqid=0, 
          callContext=0xd308e80)
          at /home/casey/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:922
      

        Attachments

          Activity

            People

            • Assignee:
              caseyc casey
              Reporter:
              caseyc casey
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: