[IMPALA-4800] apparent crash during stress run

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: crash stress
• Target Version: Impala 2.9.0
• Assignee: Unassigned
• Reporter: Michael Brown
• Created: 23/Jan/17 17:45
• Updated: 23/Jan/17 19:36
• Resolved: 23/Jan/17 19:32

Description

During a stress run over the weekend, there was a failure with printed backtrace.

[...skipped...]
#5  <signal handler called>
#6  0x0000000001351d0b in base::subtle::Acquire_Load (ptr=0x18) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/gutil/atomicops-internals-x86.h:320
#7  0x0000000001352124 in Load (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/common/atomic.h:81
#8  impala::RuntimeProfile::HighWaterMarkCounter::current_value (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:111
#9  0x0000000001352212 in impala::MemTracker::consumption (this=0x19c6e8f20) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/runtime/mem-tracker.h:286
#10 0x00000000019d2b49 in boost::_mfi::cmf0<long, impala::MemTracker>::call<impala::MemTracker* const> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:105
#11 0x00000000019d2991 in boost::_mfi::cmf0<long, impala::MemTracker>::operator()<impala::MemTracker*> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:115
#12 0x00000000019d2599 in boost::_bi::list1<boost::_bi::value<impala::MemTracker*> >::operator()<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list0> (this=0x7fd46815eca8, f=..., a=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:243
#13 0x00000000019d232d in boost::_bi::bind_t<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list1<boost::_bi::value<impala::MemTracker*> > >::operator() (this=0x7fd46815ec98) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
#14 0x00000000019d210f in boost::detail::function::function_obj_invoker0<boost::_bi::bind_t<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list1<boost::_bi::value<impala::MemTracker*> > >, long>::invoke (function_obj_ptr=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:132
#15 0x00000000015a01c4 in boost::function0<long>::operator() (this=0x7fd46815ec90) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
#16 0x000000000159fe53 in impala::RuntimeProfile::TimeSeriesCounter::AddSample (this=0x7fd46815ec80, ms_elapsed=500) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:342
#17 0x000000000159f879 in impala::PeriodicCounterUpdater::UpdateLoop (this=0x3f3da00) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/periodic-counter-updater.cc:185
#18 0x00000000015a9acf in boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>::operator() (this=0x4e7a1b8, p=0x3f3da00) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:49
#19 0x00000000015a9a32 in boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> >::operator()<boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list0> (this=0x4e7a1c8, f=..., a=...) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:253
#20 0x00000000015a99dd in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> > >::operator() (this=0x4e7a1b8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
#21 0x00000000015a99a0 in boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> > > >::run (this=0x4e7a000) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
#22 0x0000000001a3cdaa in thread_proxy ()
#23 0x00000039c68079d1 in start_thread () from /lib64/libpthread.so.0
#24 0x00000039c64e88fd in clone () from /lib64/libc.so.6

Unfortunately a different run blew away the logs/minidumps before they could be collected. Also, a subsequent run passed on the same Git hash (IMPALA-4775: minor adjustments to python test infra logging). I am re-running the stress test repeatedly to see if I can reproduce this problem.

Issue Links

• duplicates: IMPALA-4801 Heap use-after-free in expr-test