Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-6

      Description

      MemTracker::EnableReservationReporting() can run concurrently with MemTracker::LogUsage(). There is no synchronisation between the two.

      I saw a crash with this stack on a development branch, which I believe is caused by this:

      (gdb) bt
      #0  0x00000032836328e5 in raise () from sysroot/lib64/libc.so.6
      #1  0x00000032836340c5 in abort () from sysroot/lib64/libc.so.6
      #2  0x00007f2f0b8f7c55 in os::abort(bool) () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #3  0x00007f2f0ba79cd7 in VMError::report_and_die() () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #4  0x00007f2f0b8fcb6f in JVM_handle_linux_signal () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00007f2f0a75ee50 in vtable for __cxxabiv1::__si_class_type_info () from sysroot/data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6
      #7  0x00000000013ac8de in impala::MemTracker::LogUsage (this=0x5be3c80, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:253
      #8  0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #9  0x00000000013acb86 in impala::MemTracker::LogUsage (this=0xb03a6e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #10 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #11 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x157626e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #12 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #13 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x9612260, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #14 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278
      #15 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x5be3440, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269
      #16 0x00000000016b175d in impala::ExecNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/exec-node.cc:174
      #17 0x00000000017fd22f in impala::SortNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/sort-node.cc:49
      #18 0x0000000001a6a48a in impala::PlanFragmentExecutor::PrepareInternal (this=0x22dea2d0, qs=0x1c569c00, tdesc_tbl=..., fragment_ctx=..., instance_ctx=...)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:214
      #19 0x0000000001a68bcd in impala::PlanFragmentExecutor::Prepare (this=0x22dea2d0, query_state=0x1c569c00, desc_tbl=..., fragment_ctx=..., instance_ctx=...)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:99
      #20 0x0000000001a65b49 in impala::FragmentInstanceState::Exec (this=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:64
      #21 0x0000000001a71189 in impala::QueryExecMgr::ExecFInstance (this=0x8add2a0, fis=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110
      #22 0x0000000001a73fb2 in boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>::operator() (this=0x1ef7fec0, p=0x8add2a0, a1=0x22dea000)
          at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165
      #23 0x0000000001a73e3b in boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list0> (
          this=0x1ef7fed0, f=..., a=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313
      #24 0x0000000001a7396f in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator() (
          this=0x1ef7fec0) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #25 0x0000000001a7357a in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >, void>::invoke (function_obj_ptr=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
      #26 0x00000000013923d6 in boost::function0<void>::operator() (this=0x7f2e34e55c40) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
      #27 0x000000000162e6d1 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7f2e10b079a0)
          at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/thread.cc:325
      #28 0x00000000016370ac in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x171b6dc0, 
          f=@0x171b6db8: 0x162e40c <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
          at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
      #29 0x0000000001636fef in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x171b6db8) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #30 0x0000000001636fb2 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x171b6c00) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
      #31 0x0000000001aee87a in thread_proxy ()
      #32 0x0000003283a07851 in start_thread () from sysroot/lib64/libpthread.so.0
      #33 0x00000032836e894d in clone () from sysroot/lib64/libc.so.6
      

      on this line:

      int64_t reservation_limit = reservation_counters_->reservation_limit->value();
      

      This method is not currently used in query execution, but we need to fix this before switching on the buffer pool for query execution.

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5130: fix race in MemTracker::EnableReservationReporting()

        MemTracker::LogUsage() and MemTracker::EnableReservationReporting()
        could race, with LogUsage() seeing a partially-constructed
        'reservation_counters_' value and crashing.

        This patch fixes that issue by atomically swapping in
        'reservation_counters_' so that no threads see a
        partially-constructed value.

        While we're here, swap boost::mutex for the higher-performance
        SpinLock for 'child_trackers_lock_'.

        Change-Id: I2434c952d97c46040e29fca2327c244dd30599d2
        Reviewed-on: http://gerrit.cloudera.org:8080/6502
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5130 : fix race in MemTracker::EnableReservationReporting() MemTracker::LogUsage() and MemTracker::EnableReservationReporting() could race, with LogUsage() seeing a partially-constructed 'reservation_counters_' value and crashing. This patch fixes that issue by atomically swapping in 'reservation_counters_' so that no threads see a partially-constructed value. While we're here, swap boost::mutex for the higher-performance SpinLock for 'child_trackers_lock_'. Change-Id: I2434c952d97c46040e29fca2327c244dd30599d2 Reviewed-on: http://gerrit.cloudera.org:8080/6502 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development