[IMPALA-13347] TSAN failure in backend tests after IMPALA-12737

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.5.0
• Fix Version/s: Impala 4.5.0
• Component/s: Backend
• Target Version: Impala 4.5.0
• Epic Color: ghx-label-2
• Assignee: Jason Fehr
• Reporter: Michael Smith
• Created: 03/Sep/24 20:25
• Updated: 06/Sep/24 17:55
• Resolved: 06/Sep/24 17:55

Description

Impala backend tests (expr-test, session-expiry-test, internal-server-test) started detecting a thread leak in TSAN builds after IMPALA-12737 "Refactor the Workload Management Initialization Process" was merged. Example:

 WARNING: ThreadSanitizer: thread leak (pid=475)
  Thread T566 (tid=1135, finished) created by main thread at:
    #0 pthread_create <null> (unifiedbetests+0x23d7083)
    #1 boost::thread::start_thread_noexcept() <null> (unifiedbetests+0x521607d)
    #2 boost::thread::thread<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*>(void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/thread/detail/thread.hpp:424:13 (unifiedbetests+0x4f8b214)
    #3 impala::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()> const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/util/thread.cc:317:13 (unifiedbetests+0x4f8761c)
    #4 impala::Status impala::Thread::Create<boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServer>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServer*> > > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServer>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServer*> > > const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> >*, bool) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/util/thread.h:74:12 (unifiedbetests+0x4c695f2)
    #5 impala::ImpalaServer::Start(int, int, int, int) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/service/impala-server.cc:3220:5 (unifiedbetests+0x4c63e4a)
    #6 impala::InProcessImpalaServer::StartWithClientServers(int, int, int) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/testutil/in-process-servers.cc:97:3 (unifiedbetests+0x501bd3b)
    #7 impala::InProcessImpalaServer::StartWithEphemeralPorts(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, impala::InProcessImpalaServer**) /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/testutil/in-process-servers.cc:69:21 (unifiedbetests+0x501bc22)
    #8 impala::ExprTest::SetUpTestCase() /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/exprs/expr-test.cc:235:5 (unifiedbetests+0x26d42f9)
    #9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::TestSuite, void>(testing::TestSuite*, void (testing::TestSuite::*)(), char const*) <null> (unifiedbetests+0x6876a5c)
    #10 main /data/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/src/service/unified-betest-main.cc:48:10 (unifiedbetests+0x24363e0)

SUMMARY: ThreadSanitizer: thread leak (/data0/jenkins/workspace/impala-cdw-master-staging-core-tsan/repos/Impala/be/build/debug/service/unifiedbetests+0x23d7083) in __interceptor_pthread_create

Issue Links

• is related to: IMPALA-13350 Workload Management flush on interval test failed