Details

      Description

      23:45:25 14/86 Test #14: expr-test ........................***Failed  172.77 sec
      23:45:25 SLF4J: Class path contains multiple SLF4J bindings.
      23:45:25 SLF4J: Found binding in [jar:file:/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/fe/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      23:45:25 SLF4J: Found binding in [jar:file:/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      23:45:25 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      23:45:25 SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      23:45:25 Running without optimization passes.
      23:45:25 Running without codegen
      23:45:25 [==========] Running 40 tests from 1 test case.
      23:45:25 [----------] Global test environment set-up.
      23:45:25 [----------] 40 tests from ExprTest
      23:45:25 [ RUN      ] ExprTest.NullLiteral
      23:45:25 [       OK ] ExprTest.NullLiteral (10 ms)
      23:45:25 [ RUN      ] ExprTest.LiteralConstruction
      23:45:25 [       OK ] ExprTest.LiteralConstruction (12 ms)
      23:45:25 [ RUN      ] ExprTest.LiteralExprs
      23:45:25 [       OK ] ExprTest.LiteralExprs (885 ms)
      23:45:25 [ RUN      ] ExprTest.ArithmeticExprs
      23:45:25 [       OK ] ExprTest.ArithmeticExprs (4041 ms)
      23:45:25 [ RUN      ] ExprTest.DecimalArithmeticExprs
      23:45:25 [       OK ] ExprTest.DecimalArithmeticExprs (327 ms)
      23:45:25 [ RUN      ] ExprTest.BinaryPredicates
      23:45:25 [       OK ] ExprTest.BinaryPredicates (7325 ms)
      23:45:25 [ RUN      ] ExprTest.CastExprs
      23:45:25 [       OK ] ExprTest.CastExprs (5558 ms)
      23:45:25 [ RUN      ] ExprTest.CompoundPredicates
      23:45:25 [       OK ] ExprTest.CompoundPredicates (592 ms)
      23:45:25 [ RUN      ] ExprTest.IsNullPredicate
      23:45:25 [       OK ] ExprTest.IsNullPredicate (46 ms)
      23:45:25 [ RUN      ] ExprTest.LikePredicate
      23:45:25 [       OK ] ExprTest.LikePredicate (1652 ms)
      23:45:25 [ RUN      ] ExprTest.BetweenPredicate
      23:45:25 [       OK ] ExprTest.BetweenPredicate (272 ms)
      23:45:25 [ RUN      ] ExprTest.InPredicate
      23:45:25 [       OK ] ExprTest.InPredicate (1614 ms)
      23:45:25 [ RUN      ] ExprTest.StringFunctions
      23:45:25 [       OK ] ExprTest.StringFunctions (28584 ms)
      23:45:25 [ RUN      ] ExprTest.LongReverse
      23:45:25 [       OK ] ExprTest.LongReverse (24114 ms)
      23:45:25 [ RUN      ] ExprTest.StringRegexpFunctions
      23:45:25 re2/re2.cc:200: Error parsing '*': no argument for repetition operator: *
      23:45:25 [       OK ] ExprTest.StringRegexpFunctions (1292 ms)
      23:45:25 [ RUN      ] ExprTest.StringParseUrlFunction
      23:45:25 [       OK ] ExprTest.StringParseUrlFunction (1410 ms)
      23:45:25 [ RUN      ] ExprTest.UtilityFunctions
      23:45:25 [       OK ] ExprTest.UtilityFunctions (647 ms)
      23:45:25 [ RUN      ] ExprTest.SessionFunctions
      23:45:25 [       OK ] ExprTest.SessionFunctions (40 ms)
      23:45:25 [ RUN      ] ExprTest.NonFiniteFloats
      23:45:25 [       OK ] ExprTest.NonFiniteFloats (529 ms)
      23:45:25 [ RUN      ] ExprTest.InvalidFloats
      23:45:25 [       OK ] ExprTest.InvalidFloats (331 ms)
      23:45:25 [ RUN      ] ExprTest.MathTrigonometricFunctions
      23:45:25 [       OK ] ExprTest.MathTrigonometricFunctions (474 ms)
      23:45:25 [ RUN      ] ExprTest.MathConversionFunctions
      23:45:25 [       OK ] ExprTest.MathConversionFunctions (1033 ms)
      23:45:25 [ RUN      ] ExprTest.MathFunctions
      23:45:25 [       OK ] ExprTest.MathFunctions (2802 ms)
      23:45:25 [ RUN      ] ExprTest.MathRoundingFunctions
      23:45:25 [       OK ] ExprTest.MathRoundingFunctions (431 ms)
      23:45:25 [ RUN      ] ExprTest.UnaryOperators
      23:45:25 [       OK ] ExprTest.UnaryOperators (109 ms)
      23:45:25 [ RUN      ] ExprTest.TimestampFunctions
      23:45:25 [       OK ] ExprTest.TimestampFunctions (9341 ms)
      23:45:25 [ RUN      ] ExprTest.ConditionalFunctions
      23:45:25 [       OK ] ExprTest.ConditionalFunctions (2462 ms)
      23:45:25 [ RUN      ] ExprTest.ConditionalFunctionIsTrue
      23:45:25 [       OK ] ExprTest.ConditionalFunctionIsTrue (173 ms)
      23:45:25 [ RUN      ] ExprTest.ConditionalFunctionIsFalse
      23:45:25 [       OK ] ExprTest.ConditionalFunctionIsFalse (171 ms)
      23:45:25 [ RUN      ] ExprTest.ConditionalFunctionIsNotTrue
      23:45:25 [       OK ] ExprTest.ConditionalFunctionIsNotTrue (182 ms)
      23:45:25 [ RUN      ] ExprTest.ConditionalFunctionIsNotFalse
      23:45:25 [       OK ] ExprTest.ConditionalFunctionIsNotFalse (186 ms)
      23:45:25 [ RUN      ] ExprTest.ResultsLayoutTest
      23:45:25 [       OK ] ExprTest.ResultsLayoutTest (2 ms)
      23:45:25 [ RUN      ] ExprTest.DecimalFunctions
      23:45:25 [       OK ] ExprTest.DecimalFunctions (2620 ms)
      23:45:25 [ RUN      ] ExprTest.DecimalOverflowCasts
      23:45:25 [       OK ] ExprTest.DecimalOverflowCasts (378 ms)
      23:45:25 [ RUN      ] ExprTest.NullValueFunction
      23:45:25 [       OK ] ExprTest.NullValueFunction (640 ms)
      23:45:25 [ RUN      ] ExprTest.NonNullValueFunction
      23:45:25 [       OK ] ExprTest.NonNullValueFunction (623 ms)
      23:45:25 [ RUN      ] ExprTest.UdfInterfaceBuiltins
      23:45:25 [       OK ] ExprTest.UdfInterfaceBuiltins (132 ms)
      23:45:25 [ RUN      ] ExprTest.MADlib
      23:45:25 [       OK ] ExprTest.MADlib (102 ms)
      23:45:25 [ RUN      ] ExprTest.BitByteBuiltins
      23:45:25 [       OK ] ExprTest.BitByteBuiltins (1406 ms)
      23:45:25 [ RUN      ] ExprTest.UuidTest
      23:45:25 [       OK ] ExprTest.UuidTest (133 ms)
      23:45:25 [----------] 40 tests from ExprTest (102682 ms total)
      23:45:25 
      23:45:25 [----------] Global test environment tear-down
      23:45:25 [==========] 40 tests from 1 test case ran. (102682 ms total)
      23:45:25 [  PASSED  ] 40 tests.
      23:45:25 
      23:45:25 Running with codegen
      23:45:25 [==========] Running 40 tests from 1 test case.
      23:45:25 [----------] Global test environment set-up.
      23:45:25 [----------] 40 tests from ExprTest
      23:45:25 [ RUN      ] ExprTest.NullLiteral
      23:45:25 [       OK ] ExprTest.NullLiteral (2 ms)
      23:45:25 [ RUN      ] ExprTest.LiteralConstruction
      23:45:25 [       OK ] ExprTest.LiteralConstruction (3 ms)
      23:45:25 [ RUN      ] ExprTest.LiteralExprs
      23:45:25 [       OK ] ExprTest.LiteralExprs (193 ms)
      23:45:25 [ RUN      ] ExprTest.ArithmeticExprs
      23:45:25 [       OK ] ExprTest.ArithmeticExprs (3328 ms)
      23:45:25 [ RUN      ] ExprTest.DecimalArithmeticExprs
      23:45:25 [       OK ] ExprTest.DecimalArithmeticExprs (307 ms)
      23:45:25 [ RUN      ] ExprTest.BinaryPredicates
      23:45:25 [       OK ] ExprTest.BinaryPredicates (6876 ms)
      23:45:25 [ RUN      ] ExprTest.CastExprs
      23:45:25 [       OK ] ExprTest.CastExprs (5366 ms)
      23:45:25 [ RUN      ] ExprTest.CompoundPredicates
      23:45:25 [       OK ] ExprTest.CompoundPredicates (536 ms)
      23:45:25 [ RUN      ] ExprTest.IsNullPredicate
      23:45:25 [       OK ] ExprTest.IsNullPredicate (59 ms)
      23:45:25 [ RUN      ] ExprTest.LikePredicate
      23:45:25 [       OK ] ExprTest.LikePredicate (1514 ms)
      23:45:25 [ RUN      ] ExprTest.BetweenPredicate
      23:45:25 [       OK ] ExprTest.BetweenPredicate (251 ms)
      23:45:25 [ RUN      ] ExprTest.InPredicate
      23:45:25 [       OK ] ExprTest.InPredicate (1562 ms)
      23:45:25 [ RUN      ] ExprTest.StringFunctions
      23:45:25 [       OK ] ExprTest.StringFunctions (27402 ms)
      23:45:25 [ RUN      ] ExprTest.LongReverse
      23:45:25 =================================================================
      23:45:25 ==7091==ERROR: AddressSanitizer: heap-use-after-free on address 0x61300022d160 at pc 0x0000015bd076 bp 0x7fa12a3b3740 sp 0x7fa12a3b3738
      23:45:25 READ of size 8 at 0x61300022d160 thread T1
      23:45:25     #0 0x15bd075 in impala::MemTracker::consumption() const /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/mem-tracker.h:286:40
      23:45:25     #1 0x1d81ac7 in boost::_bi::bind_t<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list1<boost::_bi::value<impala::MemTracker*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #2 0x18fc465 in boost::function0<long>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14
      23:45:25     #3 0x18fca35 in impala::RuntimeProfile::TimeSeriesCounter::AddSample(int) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/runtime-profile-counters.h:342:22
      23:45:25     #4 0x18fa3b6 in impala::PeriodicCounterUpdater::UpdateLoop() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/periodic-counter-updater.cc:185:9
      23:45:25     #5 0x18ff5a7 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::PeriodicCounterUpdater>, boost::_bi::list1<boost::_bi::value<impala::PeriodicCounterUpdater*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #6 0x1deb689 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deb689)
      23:45:25     #7 0x37ebc07850 in start_thread (/lib64/libpthread.so.0+0x37ebc07850)
      23:45:25     #8 0x37eb8e894c in clone (/lib64/libc.so.6+0x37eb8e894c)
      23:45:25 
      23:45:25 0x61300022d160 is located 96 bytes inside of 328-byte region [0x61300022d100,0x61300022d248)
      23:45:25 freed by thread T28165 here:
      23:45:25     #0 0x102b7a0 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
      23:45:25     #1 0x116d212 in boost::scoped_ptr<impala::MemTracker>::reset(impala::MemTracker*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/smart_ptr/scoped_ptr.hpp:88:9
      23:45:25     #2 0x15db0d3 in impala::RuntimeState::ReleaseResources() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:300:3
      23:45:25     #3 0x1d7f237 in impala::PlanFragmentExecutor::Close() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:502:3
      23:45:25     #4 0x1d75cf1 in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:69:3
      23:45:25     #5 0x1d835ff in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:114:3
      23:45:25     #6 0x1d85537 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #7 0x1598632 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14
      23:45:25     #8 0x194e495 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
      23:45:25     #9 0x195726a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457:9
      23:45:25     #10 0x19570f7 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #11 0x1deb689 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deb689)
      23:45:25 
      23:45:25 previously allocated by thread T28165 here:
      23:45:25     #0 0x102b1a0 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
      23:45:25     #1 0x15d83a8 in impala::RuntimeState::Init() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:126:31
      23:45:25     #2 0x15d7e68 in impala::RuntimeState::RuntimeState(impala::QueryState*, impala::TPlanFragmentCtx const&, impala::TPlanFragmentInstanceCtx const&, impala::ExecEnv*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/runtime-state.cc:88:3
      23:45:25     #3 0x1d79b10 in impala::PlanFragmentExecutor::PrepareInternal(impala::QueryState*, impala::TDescriptorTable const&, impala::TPlanFragmentCtx const&, impala::TPlanFragmentInstanceCtx const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:131:11
      23:45:25     #4 0x1d79541 in impala::PlanFragmentExecutor::Prepare(impala::QueryState*, impala::TDescriptorTable const&, impala::TPlanFragmentCtx const&, impala::TPlanFragmentInstanceCtx const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:99:19
      23:45:25     #5 0x1d75c55 in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:62:7
      23:45:25     #6 0x1d835ff in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:114:3
      23:45:25     #7 0x1d85537 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #8 0x1598632 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14
      23:45:25     #9 0x194e495 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
      23:45:25     #10 0x195726a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457:9
      23:45:25     #11 0x19570f7 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16
      23:45:25     #12 0x1deb689 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deb689)
      23:45:25 
      23:45:25 Thread T1 created by T0 here:
      23:45:25     #0 0xf652f9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      23:45:25     #1 0x1deaa69 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deaa69)
      23:45:25 
      23:45:25 Thread T28165 created by T268 here:
      23:45:25     #0 0xf652f9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      23:45:25     #1 0x1deaa69 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deaa69)
      23:45:25 
      23:45:25 Thread T268 created by T121 here:
      23:45:25     #0 0xf652f9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      23:45:25     #1 0x1deaa69 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deaa69)
      23:45:25 
      23:45:25 Thread T121 created by T120 here:
      23:45:25     #0 0xf652f9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      23:45:25     #1 0x1deaa69 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deaa69)
      23:45:25 
      23:45:25 Thread T120 created by T0 here:
      23:45:25     #0 0xf652f9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      23:45:25     #1 0x1deaa69 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/exprs/expr-test+0x1deaa69)
      23:45:25 
      23:45:25 SUMMARY: AddressSanitizer: heap-use-after-free /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/mem-tracker.h:286:40 in impala::MemTracker::consumption() const
      23:45:25 Shadow bytes around the buggy address:
      23:45:25   0x0c268003d9d0: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
      23:45:25   0x0c268003d9e0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003d9f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003da00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003da10: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
      23:45:25 =>0x0c268003da20: fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd
      23:45:25   0x0c268003da30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003da40: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
      23:45:25   0x0c268003da50: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003da60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      23:45:25   0x0c268003da70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      23:45:25 Shadow byte legend (one shadow byte represents 8 application bytes):
      23:45:25   Addressable:           00
      23:45:25   Partially addressable: 01 02 03 04 05 06 07 
      23:45:25   Heap left redzone:       fa
      23:45:25   Heap right redzone:      fb
      23:45:25   Freed heap region:       fd
      23:45:25   Stack left redzone:      f1
      23:45:25   Stack mid redzone:       f2
      23:45:25   Stack right redzone:     f3
      23:45:25   Stack partial redzone:   f4
      23:45:25   Stack after return:      f5
      23:45:25   Stack use after scope:   f8
      23:45:25   Global redzone:          f9
      23:45:25   Global init order:       f6
      23:45:25   Poisoned by user:        f7
      23:45:25   Container overflow:      fc
      23:45:25   Array cookie:            ac
      23:45:25   Intra object redzone:    bb
      23:45:25   ASan internal:           fe
      23:45:25   Left alloca redzone:     ca
      23:45:25   Right alloca redzone:    cb
      23:45:25 ==7091==ABORTING
      

        Issue Links

          Activity

          Hide
          kwho Michael Ho added a comment -

          The backtrace looks eerily related to that in IMPALA-4800. Could they have the same root cause ?

          [...skipped...]
          #5  <signal handler called>
          #6  0x0000000001351d0b in base::subtle::Acquire_Load (ptr=0x18) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/gutil/atomicops-internals-x86.h:320
          #7  0x0000000001352124 in Load (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/common/atomic.h:81
          #8  impala::RuntimeProfile::HighWaterMarkCounter::current_value (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:111
          #9  0x0000000001352212 in impala::MemTracker::consumption (this=0x19c6e8f20) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/runtime/mem-tracker.h:286
          #10 0x00000000019d2b49 in boost::_mfi::cmf0<long, impala::MemTracker>::call<impala::MemTracker* const> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:105
          #11 0x00000000019d2991 in boost::_mfi::cmf0<long, impala::MemTracker>::operator()<impala::MemTracker*> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:115
          #12 0x00000000019d2599 in boost::_bi::list1<boost::_bi::value<impala::MemTracker*> >::operator()<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list0> (this=0x7fd46815eca8, f=..., a=...) at /usr/src/de
          
          Show
          kwho Michael Ho added a comment - The backtrace looks eerily related to that in IMPALA-4800 . Could they have the same root cause ? [...skipped...] #5 <signal handler called> #6 0x0000000001351d0b in base::subtle::Acquire_Load (ptr=0x18) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/gutil/atomicops-internals-x86.h:320 #7 0x0000000001352124 in Load (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/common/atomic.h:81 #8 impala::RuntimeProfile::HighWaterMarkCounter::current_value (this=0x0) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/util/runtime-profile-counters.h:111 #9 0x0000000001352212 in impala::MemTracker::consumption (this=0x19c6e8f20) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/be/src/runtime/mem-tracker.h:286 #10 0x00000000019d2b49 in boost::_mfi::cmf0<long, impala::MemTracker>::call<impala::MemTracker* const> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:105 #11 0x00000000019d2991 in boost::_mfi::cmf0<long, impala::MemTracker>::operator()<impala::MemTracker*> (this=0x7fd46815ec98, u=@0x7fd46815eca8) at /usr/src/debug/impala-2.7.0-cdh5.11.0-SNAPSHOT/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:115 #12 0x00000000019d2599 in boost::_bi::list1<boost::_bi::value<impala::MemTracker*> >::operator()<long, boost::_mfi::cmf0<long, impala::MemTracker>, boost::_bi::list0> (this=0x7fd46815eca8, f=..., a=...) at /usr/src/de
          Hide
          tarmstrong Tim Armstrong added a comment -

          Yeah I figured out what this is. RuntimeProfile does some cleanup in it's destructor, which includes unregistering some callbacks that periodically touch MemTrackers. So if the MemTracker gets destroyed before the profile, we have a problem.

          Ideally we should move that out of the RuntimeProfile destructor, but we can fix the immediate problem by fixing the lifetime of the MemTracker.

          Show
          tarmstrong Tim Armstrong added a comment - Yeah I figured out what this is. RuntimeProfile does some cleanup in it's destructor, which includes unregistering some callbacks that periodically touch MemTrackers. So if the MemTracker gets destroyed before the profile, we have a problem. Ideally we should move that out of the RuntimeProfile destructor, but we can fix the immediate problem by fixing the lifetime of the MemTracker.
          Hide
          mikesbrown Michael Brown added a comment -

          Another stress test run hit this.

          Show
          mikesbrown Michael Brown added a comment - Another stress test run hit this.
          Hide
          tarmstrong Tim Armstrong added a comment -

          IMPALA-4801: fix heap use after free for MemTracker

          This is a minimal fix for IMPALA-4801. The bug was that
          "IMPALA-4678: move query MemTracker into QueryState" changed
          the lifecycle of RuntimeState::instance_mem_tracker_ so that
          it destroyed earlier in ReleaseResources(). A periodic
          counter referencing the MemTracker was only stopped after
          ReleaseResources() was called, leaving a window for a race.

          There are various issues with the lifecycle of RuntimeProfile
          objects (see IMPALA-4802) but I chose not to tackle them in
          this patch.

          Change-Id: Iccaf6700888c333b9c93d716d395d96bc5af6631
          Reviewed-on: http://gerrit.cloudera.org:8080/5772
          Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
          Tested-by: Impala Public Jenkins

          M be/src/runtime/plan-fragment-executor.cc
          1 file changed, 5 insertions, 3 deletions

          Approvals:
          Impala Public Jenkins: Verified
          Tim Armstrong: Looks good to me, approved

          Show
          tarmstrong Tim Armstrong added a comment - IMPALA-4801 : fix heap use after free for MemTracker This is a minimal fix for IMPALA-4801 . The bug was that " IMPALA-4678 : move query MemTracker into QueryState" changed the lifecycle of RuntimeState::instance_mem_tracker_ so that it destroyed earlier in ReleaseResources(). A periodic counter referencing the MemTracker was only stopped after ReleaseResources() was called, leaving a window for a race. There are various issues with the lifecycle of RuntimeProfile objects (see IMPALA-4802 ) but I chose not to tackle them in this patch. Change-Id: Iccaf6700888c333b9c93d716d395d96bc5af6631 Reviewed-on: http://gerrit.cloudera.org:8080/5772 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins — M be/src/runtime/plan-fragment-executor.cc 1 file changed, 5 insertions , 3 deletions Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              twmarshall Thomas Tauber-Marshall
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development