[IMPALA-9941] ExprTest.CastExprs fails when running with ASAN

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: broken-build
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-11
• Assignee: Adam Tamas
• Reporter: Joe McDonnell
• Created: 10/Jul/20 01:21
• Updated: 16/Jul/20 16:02
• Resolved: 16/Jul/20 13:43

Description

I was running ASAN jobs to test the sharding fix. When running the backend tests under ASAN, ExprTest.CastExprs fails with the following message:

 

==4725==ERROR: AddressSanitizer: use-after-poison on address 0x6210004b3d27 at pc 0x000003accb3e bp 0x7fabbdb14310 sp 0x7fabbdb14308
READ of size 1 at 0x6210004b3d27 thread T29762
    #0 0x3accb3d in impala::datetime_parse_util::SimpleDateFormatTokenizer::GetDefaultFormatContext(char const*, int, bool, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/datetime-simple-date-format-parser.cc:345:24
    #1 0x3c7053d in impala::TimestampParser::ParseSimpleDateFormat(char const*, int, boost::gregorian::date*, boost::posix_time::time_duration*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/timestamp-parse-util.cc:109:7
    #2 0x3c73f9b in impala::TimestampValue::ParseSimpleDateFormat(char const*, int) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/timestamp-value.cc:53:18
    #3 0x307d292 in impala::CastFunctions::CastToTimestampVal(impala_udf::FunctionContext*, impala_udf::StringVal const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/cast-functions-ir.cc:321:10
    #4 0x3045142 in impala_udf::TimestampVal impala::ScalarFnCall::InterpretEval<impala_udf::TimestampVal>(impala::ScalarExprEvaluator*, impala::TupleRow const*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-fn-call.cc:457:7
    #5 0x3010e33 in impala::ScalarFnCall::GetTimestampValInterpreted(impala::ScalarExprEvaluator*, impala::TupleRow const*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-fn-call.cc:525:1
    #6 0x2fad98e in impala::ScalarExpr::GetTimestampVal(impala::ScalarExprEvaluator*, impala::TupleRow const*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr.inline.h:60:1
    #7 0x2faa522 in impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, impala::TupleRow const*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:318:41
    #8 0x3c7c6bf in void impala::Tuple::MaterializeExprs<false, false>(impala::TupleRow*, impala::TupleDescriptor const&, impala::ScalarExprEvaluator* const*, impala::MemPool*, impala::StringValue**, int*, int*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/tuple.cc:234:27
    #9 0x2e26840 in void impala::Tuple::MaterializeExprs<false, false>(impala::TupleRow*, impala::TupleDescriptor const&, std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> > const&, impala::MemPool*, std::vector<impala::StringValue*, std::allocator<impala::StringValue*> >*, int*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/tuple.h:191:5
    #10 0x2e339d6 in impala::UnionNode::MaterializeExprs(std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> > const&, impala::TupleRow*, unsigned char*, impala::RowBatch*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/union-node-ir.cc:29:14
    #11 0x2e2d5c1 in impala::UnionNode::GetNextConst(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/union-node.cc:289:5
    #12 0x2e2ded5 in impala::UnionNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/union-node.cc:323:5
    #13 0x3b50780 in impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:416:7
    #14 0x3b4c8e2 in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:106:14
    #15 0x3be69ce in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:815:24
    #16 0x3922446 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #17 0x4ab5a89 in impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3
    #18 0x4ac10d8 in void boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
    #19 0x4ac0f2b in boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #20 0x4e929d1 in thread_proxy (/data0/jenkins/workspace/impala-private-parameterized/repos/Impala/be/build/debug/service/unifiedbetests+0x4e929d1)
    #21 0x7fb4b8ca9e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
    #22 0x7fb4b581e34c in __clone (/lib64/libc.so.6+0xf834c)

0x6210004b3d27 is located 39 bytes inside of 4096-byte region [0x6210004b3d00,0x6210004b4d00)
allocated by thread T29762 here:
    #0 0x1df7dc0 in malloc /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67
    #1 0x3bd1353 in impala::MemPool::FindChunk(long, bool) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/mem-pool.cc:132:45
    #2 0x22f37b6 in unsigned char* impala::MemPool::Allocate<true>(long, int) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/mem-pool.h:295:9
    #3 0x22f37b6 in impala::MemPool::TryAllocateAligned(long, int) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/mem-pool.h:130
    #4 0x2f8a087 in impala::AllocateAnyVal(impala::RuntimeState*, impala::MemPool*, impala::ColumnType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, impala_udf::AnyVal**) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/anyval-util.cc:34:39
    #5 0x300c720 in impala::ScalarFnCall::OpenEvaluator(impala_udf::FunctionContext::FunctionStateScope, impala::RuntimeState*, impala::ScalarExprEvaluator*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-fn-call.cc:152:5
    #6 0x2fa8266 in impala::ScalarExprEvaluator::Open(impala::RuntimeState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:141:16
    #7 0x2fa8486 in impala::ScalarExprEvaluator::Open(std::vector<impala::ScalarExprEvaluator*, std::allocator<impala::ScalarExprEvaluator*> > const&, impala::RuntimeState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:146:42
    #8 0x2e2a18b in impala::UnionNode::Open(impala::RuntimeState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/union-node.cc:169:5
    #9 0x3b4fd68 in impala::FragmentInstanceState::Open() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:396:5
    #10 0x3b4c735 in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:97:12
    #11 0x3be69ce in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:815:24
    #12 0x3922446 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #13 0x4ab5a89 in impala::Thread::SuperviseThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3
    #14 0x4ac10d8 in void boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
    #15 0x4ac0f2b in boost::_bi::bind_t<void, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #16 0x4e929d1 in thread_proxy (/data0/jenkins/workspace/impala-private-parameterized/repos/Impala/be/build/debug/service/unifiedbetests+0x4e929d1)

It reproduces on an ASAN build (buildall.sh -asan -skiptests) with the following command:

 

 

be/build/latest/exprs/expr-test --gtest_filter=Instantiations/ExprTest.CastExprs*

This is related to the fix for "IMPALA-9531: Dropped support for dateless timestamps". When I revert that change, the test passes.

 

Issue Links

• is caused by: IMPALA-9531 Drop support for "dateless timestamps"
• is related to: IMPALA-9961 Invalid memory access in SimpleDataFormatTokenizer