[KUDU-2212] TSAN "destroy of locked mutex" failure in kudu-admin-test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: test
• Target Version/s: Backlog
• Code Review: https://gerrit.cloudera.org/#/c/13763/
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 08/Nov/17 19:42
• Updated: 17/Aug/19 00:22
• Resolved: 17/Aug/19 00:22

Description

admin cli test is flaky with:
WARNING: ThreadSanitizer: destroy of a locked mutex (pid=16401)
#0 pthread_rwlock_destroy /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1214 (kudu+0x49c866)
#1 glog_internal_namespace_::Mutex::~Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 (libglog.so.0+0x15878)
#2 at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:375 (kudu+0x4706a3)

and:
#0 memcpy /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:655 (kudu+0x49344c)
#1 std::_1::char_traits<char>::copy(char*, char const*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/_string:223:50 (kudu+0x51285e)
#2 std::_1::basic_string<char, std::1::char_traits<char>, std::1::allocator<char> >::_init(char const*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/string:1538:5 (libkudu_util.so+0x13c118)
#3 std::_1::basic_string<char, std::1::char_traits<char>, std::_1::allocator<char> >::basic_string(char const*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/string:1547 (libkudu_util.so+0x13c118)
#4 kudu::flag_tags_internal::FlagTagger::FlagTagger(char const*, char const*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/flag_tags.cc:76 (libkudu_util.so+0x13c118)
#5 __cxx_global_var_init.8 /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/rpc_server.cc:54:1 (libserver_process.so+0x5f7c8)
#6 GLOBAL_sub_I_rpc_server.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/server/rpc_server.cc (libserver_process.so+0x5fcfc)
#7 _dl_init_internal <null> (ld-linux-x86-64.so.2+0xe64e)

Location is global 'google::log_mutex' of size 64 at 0x7ff79ecd3038 (libglog.so.0+0x00000022b038)

Mutex M115 (0x7ff79ecd3038) created at:
#0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1205 (kudu+0x49ca9c)
#1 glog_internal_namespace_::Mutex::Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:247:19 (libglog.so.0+0xc5f3)
#2 __cxx_global_var_init.130 /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/logging.cc:372 (libglog.so.0+0xc5f3)
#3 GLOBAL_sub_I_logging.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/logging.cc (libglog.so.0+0xc5f3)
#4 _dl_init_internal <null> (ld-linux-x86-64.so.2+0xe64e)

Issue Links

• is duplicated by: KUDU-2916 Admin.TestDumpMemTrackers is flaky in tsan
• is duplicated by: KUDU-2765 tsan failure in ToolTest.TestLoadgenAutoFlushBackgroundRandom