[IMPALA-6488] Crash in LibCache::GetCacheEntryInternal()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0, Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Backend
• Labels: broken-build crash flaky
• Epic Color: ghx-label-8
• Assignee: Vuk Ercegovac
• Reporter: Vuk Ercegovac
• Created: 07/Feb/18 18:55
• Updated: 23/Mar/18 05:16
• Resolved: 15/Mar/18 20:35

Description

Recent runs show a crash in LibCache::GetCacheEntryInternal(). Example stack trace of a crashing thread: 

 

#0  0x00007f0f6ab998e5 in raise () from /lib64/libc.so.6
#1  0x00007f0f6ab9b0c5 in abort () from /lib64/libc.so.6
#2  0x00007f0f6b61107d in __gnu_cxx::__verbose_terminate_handler() () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007f0f6b60f0e6 in __cxxabiv1::__terminate(void (*)()) () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007f0f6b60f131 in std::terminate() () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x00007f0f6b60f348 in __cxa_throw () at ../../../../gcc-4.9.2/libstdc++-v3/libsupc++/eh_throw.cc:87
#6  0x0000000000943659 in boost::throw_exception<boost::lock_error> (e=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/throw_exception.hpp:69
#7  0x0000000000b303fa in lock (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/pthread/mutex.hpp:119
#8  lock (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/lock_types.hpp:346
#9  unique_lock (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/lock_types.hpp:124
#10 impala::LibCache::GetCacheEntryInternal (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/runtime/lib-cache.cc:357
#11 0x0000000000b30844 in impala::LibCache::GetCacheEntry (this=0x4a77e30, hdfs_lib_file=..., type=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/runtime/lib-cache.cc:295
#12 0x0000000000b31432 in impala::LibCache::GetLocalPath (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/runtime/lib-cache.cc:192
#13 0x000000000106c867 in impala::HiveUdfCall::OpenEvaluator (this=0x13757680, scope=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/exprs/hive-udf-call.cc:204
#14 0x0000000001070ff8 in impala::ScalarExpr::OpenEvaluator (this=0x11046e00, scope=impala_udf::FunctionContext::FRAGMENT_LOCAL, state=0x7f0e8bbcbaf0, eval=0x33781200) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/exprs/scalar-expr.cc:200
#15 0x0000000001079ec1 in impala::ScalarFnCall::OpenEvaluator (this=0x11046e00, scope=impala_udf::FunctionContext::FRAGMENT_LOCAL, state=0x7f0e8bbcbaf0, eval=0x33781200) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/exprs/scalar-fn-call.cc:154
#16 0x0000000000969c27 in impala::ScalarExprEvaluator::Open (this=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:137
#17 0x0000000000bcb5fd in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0x1b7f11e0, caller_class=) at /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/be/src/service/fe-support.cc:233
#18 0x00007f0f51f4c79d in ?? ()
#19 0x000000066afffb48 in ?? ()
#20 0x000000066b001930 in ?? ()
#21 0x0000000000000000 in ?? ()

Issue Links

• blocks: IMPALA-6215 Race between lib_cache and java udf class loading
• breaks: IMPALA-6722 Local build failing due to missing libTestUdfs.so