[IMPALA-9730] TSAN data race in RuntimeFilterBank::CancelLocked() - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: Impala 4.0.0
Component/s: None
Labels:
None

Epic Color:
ghx-label-2

Description

I started seeing the issue while working on ~~IMPALA-9199~~. I don't see a reason why ~~IMPALA-9199~~ would have started causing this though. The failure comes up in expr-test which should be unrelated to query retries.

WARNING: ThreadSanitizer: data race (pid=29921)
  Read of size 1 at 0x7b340005c5c8 by thread T370:
    #0 impala::RuntimeFilterBank::CancelLocked() /home/stakiar/Impala/be/src/runtime/runtime-filter-bank.cc:435 (unifiedbetests+0x2da33cb)
    #1 impala::RuntimeFilterBank::Close() /home/stakiar/Impala/be/src/runtime/runtime-filter-bank.cc:459 (unifiedbetests+0x2da371a)
    #2 impala::QueryState::ReleaseBackendResources() /home/stakiar/Impala/be/src/runtime/query-state.cc:115 (unifiedbetests+0x2d7bcbb)
    #3 impala::QueryState::ReleaseBackendResourceRefcount() /home/stakiar/Impala/be/src/runtime/query-state.cc:731 (unifiedbetests+0x2d8084f)
    #4 impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*) /home/stakiar/Impala/be/src/runtime/query-exec-mgr.cc:157 (unifiedbetests+0x2df74dd)
    #5 boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*) const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/mem
_fn_template.hpp:165 (unifiedbetests+0x2df87bd)
    #6 void boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::
list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:319 (
unifiedbetests+0x2df86fd)
    #7 boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::QueryState*> > >::o
perator()() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x2df8663)
    #8 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExec
Mgr*>, boost::_bi::value<impala::QueryState*> > >, void>::invoke(boost::detail::function::function_buffer&) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
(unifiedbetests+0x2df8401)
    #9 boost::function0<void>::operator()() const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770 (unifiedbetests+0x2bc2061)
    #10 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /home/stakiar/Impa
la/be/src/util/thread.cc:360 (unifiedbetests+0x3533c86)
    #11 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<
impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Prom
iseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseM
ode)0>*), boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 (unifiedbetests+0x353be7c)
    #12 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::lis
t5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impal
a::PromiseMode)0>*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x353bd93)
    #13 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Promis
eMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<i
mpala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 (unifiedbetests+0x353ba80)
    #14 thread_proxy thread.cpp:? (unifiedbetests+0x37c7869)

  Previous write of size 1 at 0x7b340005c5c8 by thread T113:
    #0 impala::RuntimeFilterBank::CancelLocked() /home/stakiar/Impala/be/src/runtime/runtime-filter-bank.cc:440 (unifiedbetests+0x2da3475)
    #1 impala::RuntimeFilterBank::Cancel() /home/stakiar/Impala/be/src/runtime/runtime-filter-bank.cc:431 (unifiedbetests+0x2da330a)
    #2 impala::QueryState::Cancel() /home/stakiar/Impala/be/src/runtime/query-state.cc:768 (unifiedbetests+0x2d7f54f)
    #3 impala::ControlService::CancelQueryFInstances(impala::CancelQueryFInstancesRequestPB const*, impala::CancelQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) /home/stakiar/Impala/be/src/service/co
ntrol-service.cc:234 (unifiedbetests+0x2eee30b)
    #4 operator() /home/stakiar/Impala/be/generated-sources/gen-cpp/control_service.service.cc:110 (unifiedbetests+0x2fa3974)
    #5 std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> cons
t&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /home/stakiar/Impala/toolc
hain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039 (unifiedbetests+0x2fa375e)
    #6 std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcCo
ntext*) const /home/stakiar/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439 (unifiedbetests+0x36940e8)
    #7 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/stakiar/Impala/be/src/kudu/rpc/service_if.cc:139 (unifiedbetests+0x3693a7e)
    #8 impala::ImpalaServicePool::RunThread() /home/stakiar/Impala/be/src/rpc/impala-service-pool.cc:272 (unifiedbetests+0x36ab83f)
    #9 boost::_mfi::mf0<void, impala::ImpalaServicePool>::operator()(impala::ImpalaServicePool*) const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49 (unifiedbete
sts+0x36b0856)
    #10 void boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >::operator()<boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<vo
id, impala::ImpalaServicePool>&, boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:259 (unifiedbetests+0x36b07aa)
    #11 boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1
.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x36b0733)
    #12 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >,
 void>::invoke(boost::detail::function::function_buffer&) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159 (unifiedbetests+0x36b0529)
    #13 boost::function0<void>::operator()() const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770 (unifiedbetests+0x2bc2061)
    #14 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /home/stakiar/Impa
la/be/src/util/thread.cc:360 (unifiedbetests+0x3533c86)
    #15 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<
impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Prom
iseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseM
ode)0>*), boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 (unifiedbetests+0x353be7c)
    #16 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::lis
t5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impal
a::PromiseMode)0>*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x353bd93)
    #17 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Promis
eMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<i
mpala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 (unifiedbetests+0x353ba80)
    #18 thread_proxy thread.cpp:? (unifiedbetests+0x37c7869)

  Location is heap block of size 200 at 0x7b340005c520 allocated by thread T113:
    #0 operator new(unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:44 (discriminator 2) (unifiedbetests+0x1c9c0b3)
    #1 impala::QueryState::InitFilterBank() /home/stakiar/Impala/be/src/runtime/query-state.cc:341 (unifiedbetests+0x2d7d501)
    #2 impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB const*, impala::TExecPlanFragmentInfo const&) /home/stakiar/Impala/be/src/runtime/query-state.cc:232 (unifiedbetests+0x2d7c77c)
    #3 impala::QueryExecMgr::StartQuery(impala::ExecQueryFInstancesRequestPB const*, impala::TQueryCtx const&, impala::TExecPlanFragmentInfo const&) /home/stakiar/Impala/be/src/runtime/query-exec-mgr.cc:6
0 (unifiedbetests+0x2df6817)
    #4 impala::ControlService::ExecQueryFInstances(impala::ExecQueryFInstancesRequestPB const*, impala::ExecQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) /home/stakiar/Impala/be/src/service/control-
service.cc:157 (unifiedbetests+0x2eed27e)
    #5 operator() /home/stakiar/Impala/be/generated-sources/gen-cpp/control_service.service.cc:70 (unifiedbetests+0x2fa2474)
    #6 std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> cons
t&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /home/stakiar/Impala/toolc
hain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039 (unifiedbetests+0x2fa225e)
    #7 std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcCo
ntext*) const /home/stakiar/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439 (unifiedbetests+0x36940e8)
    #8 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/stakiar/Impala/be/src/kudu/rpc/service_if.cc:139 (unifiedbetests+0x3693a7e)
    #9 impala::ImpalaServicePool::RunThread() /home/stakiar/Impala/be/src/rpc/impala-service-pool.cc:272 (unifiedbetests+0x36ab83f)
    #10 boost::_mfi::mf0<void, impala::ImpalaServicePool>::operator()(impala::ImpalaServicePool*) const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49 (unifiedbet
ests+0x36b0856)
    #11 void boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >::operator()<boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<v$
id, impala::ImpalaServicePool>&, boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:259 (unifiedbetests+0x36b07aa)
    #12 boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1
.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x36b0733)
    #13 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >,
 void>::invoke(boost::detail::function::function_buffer&) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159 (unifiedbetests+0x36b0529)
    #14 boost::function0<void>::operator()() const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770 (unifiedbetests+0x2bc2061)
    #15 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /home/stakiar/Impa
la/be/src/util/thread.cc:360 (unifiedbetests+0x3533c86)
    #16 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<
impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Prom
iseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseM
ode)0>*), boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 (unifiedbetests+0x353be7c)
    #17 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::lis
t5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impal
a::PromiseMode)0>*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x353bd93)
    #18 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Promis
eMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<i
mpala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 (unifiedbetests+0x353ba80)
    #19 thread_proxy thread.cpp:? (unifiedbetests+0x37c7869)

  Thread T370 (tid=19643, running) created by thread T113 at:
    #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (unifiedbetests+0x1c33a4b)
    #1 boost::thread::start_thread_noexcept() ??:? (unifiedbetests+0x37c6bf9)
    #2 thread<void (*)(const std::basic_string<char> &, const std::basic_string<char> &, boost::function<void ()>, const impala::ThreadDebugInfo *, impala::Promise<long, impala::PromiseMode::SINGLE_PRODUC
ER> *), std::basic_string<char>, std::basic_string<char>, boost::function<void ()>, impala::ThreadDebugInfo *, impala::Promise<long, impala::PromiseMode::SINGLE_PRODUCER> *> /home/stakiar/Impala/toolchain
/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420 (unifiedbetests+0x353645b)
    #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/stakiar/Impa
la/be/src/util/thread.cc:317 (unifiedbetests+0x3533625)
    #4 impala::Status impala::Thread::Create<void (impala::QueryExecMgr::*)(impala::QueryState*), impala::QueryExecMgr*, impala::QueryState*>(std::string const&, std::string const&, void (impala::QueryExe
cMgr::* const&)(impala::QueryState*), impala::QueryExecMgr* const&, impala::QueryState* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/stakiar/Impala/be/src/ut
il/thread.h:89 (unifiedbetests+0x2df7aa7)
    #5 impala::QueryExecMgr::StartQuery(impala::ExecQueryFInstancesRequestPB const*, impala::TQueryCtx const&, impala::TExecPlanFragmentInfo const&) /home/stakiar/Impala/be/src/runtime/query-exec-mgr.cc:6
9 (unifiedbetests+0x2df6992)
    #6 impala::ControlService::ExecQueryFInstances(impala::ExecQueryFInstancesRequestPB const*, impala::ExecQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) /home/stakiar/Impala/be/src/service/control-
service.cc:157 (unifiedbetests+0x2eed27e)
    #7 operator() /home/stakiar/Impala/be/generated-sources/gen-cpp/control_service.service.cc:70 (unifiedbetests+0x2fa2474)
    #8 std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> cons
t&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /home/stakiar/Impala/toolc
hain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039 (unifiedbetests+0x2fa225e)
    #9 std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcC$
ntext*) const /home/stakiar/Impala/toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439 (unifiedbetests+0x36940e8)
    #10 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/stakiar/Impala/be/src/kudu/rpc/service_if.cc:139 (unifiedbetests+0x3693a7e)
    #11 impala::ImpalaServicePool::RunThread() /home/stakiar/Impala/be/src/rpc/impala-service-pool.cc:272 (unifiedbetests+0x36ab83f)
    #12 boost::_mfi::mf0<void, impala::ImpalaServicePool>::operator()(impala::ImpalaServicePool*) const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49 (unifiedbet
ests+0x36b0856)
    #13 void boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> >::operator()<boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<vo
id, impala::ImpalaServicePool>&, boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:259 (unifiedbetests+0x36b07aa)
    #14 boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1
.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x36b0733)
    #15 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >,
 void>::invoke(boost::detail::function::function_buffer&) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159 (unifiedbetests+0x36b0529)
    #16 boost::function0<void>::operator()() const /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770 (unifiedbetests+0x2bc2061)
    #17 impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /home/stakiar/Impa
la/be/src/util/thread.cc:360 (unifiedbetests+0x3533c86)
    #18 void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<
impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Prom
iseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseM
ode)0>*), boost::_bi::list0&, int) /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531 (unifiedbetests+0x353be7c)
    #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::lis
t5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impal
a::PromiseMode)0>*> > >::operator()() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222 (unifiedbetests+0x353bd93)
    #20 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::Promis
eMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<i
mpala::Promise<long, (impala::PromiseMode)0>*> > > >::run() /home/stakiar/Impala/toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116 (unifiedbetests+0x353ba80)
    #21 thread_proxy thread.cpp:? (unifiedbetests+0x37c7869)  Thread T113 (tid=30180, running) created by main thread at:
    #0 pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 (unifiedbetests+0x1c33a4b)
    #1 boost::thread::start_thread_noexcept() ??:? (unifiedbetests+0x37c6bf9)
    #2 thread<void (*)(const std::basic_string<char> &, const std::basic_string<char> &, boost::function<void ()>, const impala::ThreadDebugInfo *, impala::Promise<long, impala::PromiseMode::SINGLE_PRODUC
ER> *), std::basic_string<char>, std::basic_string<char>, boost::function<void ()>, impala::ThreadDebugInfo *, impala::Promise<long, impala::PromiseMode::SINGLE_PRODUCER> *> /home/stakiar/Impala/toolchain
/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420 (unifiedbetests+0x353645b)
    #3 impala::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/stakiar/Impa
la/be/src/util/thread.cc:317 (unifiedbetests+0x3533625)
    #4 impala::Status impala::Thread::Create<void (impala::ImpalaServicePool::*)(), impala::ImpalaServicePool*>(std::string const&, std::string const&, void (impala::ImpalaServicePool::* const&)(), impala
::ImpalaServicePool* const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /home/stakiar/Impala/be/src/util/thread.h:81 (unifiedbetests+0x36ad877)
    #5 impala::ImpalaServicePool::Init(int) /home/stakiar/Impala/be/src/rpc/impala-service-pool.cc:92 (unifiedbetests+0x36ab585)
    #6 impala::RpcMgr::RegisterService(int, int, kudu::rpc::GeneratedServiceIf*, impala::MemTracker*) /home/stakiar/Impala/be/src/rpc/rpc-mgr.cc:163 (unifiedbetests+0x3694c54)
    #7 impala::ControlService::Init() /home/stakiar/Impala/be/src/service/control-service.cc:80 (unifiedbetests+0x2eeca70)
    #8 impala::ExecEnv::Init() /home/stakiar/Impala/be/src/runtime/exec-env.cc:387 (unifiedbetests+0x2d07404)
    #9 impala::InProcessImpalaServer::StartWithClientServers(int, int, int) /home/stakiar/Impala/be/src/testutil/in-process-servers.cc:81 (unifiedbetests+0x30270f0)
    #10 impala::InProcessImpalaServer::StartWithEphemeralPorts(std::string const&, int, impala::InProcessImpalaServer**) /home/stakiar/Impala/be/src/testutil/in-process-servers.cc:60 (unifiedbetests+0x302
7079)
    #11 impala::ExprTest::SetUpTestCase() /home/stakiar/Impala/be/src/exprs/expr-test.cc:231 (unifiedbetests+0x1e17b78)
    #12 testing::TestCase::Run() ??:? (unifiedbetests+0x49a5832)
    #13 __libc_start_main ??:? (libc.so.6+0x2082f)SUMMARY: ThreadSanitizer: data race /home/stakiar/Impala/be/src/runtime/runtime-filter-bank.cc:435 in impala::RuntimeFilterBank::CancelLocked()

I investigated the issue a bit, and it looks like there is a (seemingly harmless) data race in the cancellation code path of RuntimeFilterBank that can be triggered if no filters are actually registered.

The cancellation code in RuntimeFilterBank is as follows:

  vector<unique_lock<SpinLock>> RuntimeFilterBank::LockAllFilters() {
    vector<unique_lock<SpinLock>> locks;
    for (auto& entry : filters_) locks.emplace_back(entry.second->lock);
    return locks;
  }

  void RuntimeFilterBank::Cancel() {
    auto all_locks = LockAllFilters();
    CancelLocked();
  }

  void RuntimeFilterBank::CancelLocked() {
    if (cancelled_) return;
    // Cancel all filters that a thread might be waiting on.
    for (auto& entry : filters_) {
    ¦ if (entry.second->consumed_filter != nullptr) entry.second->consumed_filter->Cancel();
    }
    cancelled_ = true;
  }

The issue is that if filters_ is empty, then no locks get acquired, which causes a race on cancelled_.

Attachments

Issue Links

is caused by

IMPALA-9380 Serialize query profile asynchronously

Resolved

is duplicated by

IMPALA-9742 Data race in RuntimeFilterBank::CancelLocked in TSAN builds

Resolved

Activity

People

Assignee:: Tim Armstrong

Reporter:: Sahil Takiar

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 07/May/20 00:07

Updated:: 19/May/20 02:49

Resolved:: 19/May/20 02:49