Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.9.0
-
None
-
ghx-label-6
Description
MemTracker::EnableReservationReporting() can run concurrently with MemTracker::LogUsage(). There is no synchronisation between the two.
I saw a crash with this stack on a development branch, which I believe is caused by this:
(gdb) bt #0 0x00000032836328e5 in raise () from sysroot/lib64/libc.so.6 #1 0x00000032836340c5 in abort () from sysroot/lib64/libc.so.6 #2 0x00007f2f0b8f7c55 in os::abort(bool) () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so #3 0x00007f2f0ba79cd7 in VMError::report_and_die() () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so #4 0x00007f2f0b8fcb6f in JVM_handle_linux_signal () from sysroot/opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so #5 <signal handler called> #6 0x00007f2f0a75ee50 in vtable for __cxxabiv1::__si_class_type_info () from sysroot/data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6 #7 0x00000000013ac8de in impala::MemTracker::LogUsage (this=0x5be3c80, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:253 #8 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278 #9 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0xb03a6e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269 #10 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278 #11 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x157626e0, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269 #12 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278 #13 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x9612260, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269 #14 0x00000000013acdab in impala::MemTracker::LogUsage (prefix=..., trackers=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:278 #15 0x00000000013acb86 in impala::MemTracker::LogUsage (this=0x5be3440, prefix=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/mem-tracker.cc:269 #16 0x00000000016b175d in impala::ExecNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/exec-node.cc:174 #17 0x00000000017fd22f in impala::SortNode::Prepare (this=0x14d55a20, state=0xb056000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/exec/sort-node.cc:49 #18 0x0000000001a6a48a in impala::PlanFragmentExecutor::PrepareInternal (this=0x22dea2d0, qs=0x1c569c00, tdesc_tbl=..., fragment_ctx=..., instance_ctx=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:214 #19 0x0000000001a68bcd in impala::PlanFragmentExecutor::Prepare (this=0x22dea2d0, query_state=0x1c569c00, desc_tbl=..., fragment_ctx=..., instance_ctx=...) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:99 #20 0x0000000001a65b49 in impala::FragmentInstanceState::Exec (this=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:64 #21 0x0000000001a71189 in impala::QueryExecMgr::ExecFInstance (this=0x8add2a0, fis=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110 #22 0x0000000001a73fb2 in boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>::operator() (this=0x1ef7fec0, p=0x8add2a0, a1=0x22dea000) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165 #23 0x0000000001a73e3b in boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list0> ( this=0x1ef7fed0, f=..., a=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313 #24 0x0000000001a7396f in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator() ( this=0x1ef7fec0) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20 #25 0x0000000001a7357a in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >, void>::invoke (function_obj_ptr=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153 #26 0x00000000013923d6 in boost::function0<void>::operator() (this=0x7f2e34e55c40) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767 #27 0x000000000162e6d1 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7f2e10b079a0) at /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/thread.cc:325 #28 0x00000000016370ac in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x171b6dc0, f=@0x171b6db8: 0x162e40c <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457 #29 0x0000000001636fef in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x171b6db8) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20 #30 0x0000000001636fb2 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x171b6c00) at /data/jenkins/workspace/impala-private-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116 #31 0x0000000001aee87a in thread_proxy () #32 0x0000003283a07851 in start_thread () from sysroot/lib64/libpthread.so.0 #33 0x00000032836e894d in clone () from sysroot/lib64/libc.so.6
on this line:
int64_t reservation_limit = reservation_counters_->reservation_limit->value();
This method is not currently used in query execution, but we need to fix this before switching on the buffer pool for query execution.