[KUDU-1507] MemTracker CHECK failure under load in DeleteTableTest.TestDeleteTableWithConcurrentWrites

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.9.0
• Fix Version/s: n/a
• Component/s: tablet
• Assignee: Unassigned
• Reporter: Mike Percy
• Created: 29/Jun/16 01:55
• Updated: 30/Jun/16 05:35
• Resolved: 30/Jun/16 05:35

Description

I looped DeleteTableTest.TestDeleteTableWithConcurrentWrites 500x in TSAN and was able to trigger the following CHECK / test failure:

 4829 F0628 23:54:51.354387 28898 mem_tracker.cc:186] Check failed: consumption() == 0 Memory tracker txn_tracker->tablet-a16b944a988b4388b55f8396b18aa8b9->server->root has unreleased consumption 2400
 4830 *** Check failure stack trace: ***
 4831     @     0x7f378e217304  google::LogMessage::SendToLog() at ??:0
 4832     @     0x7f378e217800  google::LogMessage::Flush() at ??:0
 4833     @     0x7f378e21b432  google::LogMessageFatal::~LogMessageFatal() at ??:0
 4834     @     0x7f378ef05098  kudu::MemTracker::~MemTracker() at ??:0
 4835     @     0x7f378ef087ec  std::_Sp_counted_ptr<>::_M_dispose() at ??:0
 4836     @     0x7f3795d8cc01  std::_Sp_counted_base<>::_M_release() at ??:0
 4837     @     0x7f3795d8cbb7  std::__shared_count<>::~__shared_count() at ??:0
 4838     @     0x7f3795dc0f5e  std::__shared_ptr<>::~__shared_ptr() at ??:0
 4839     @     0x7f379486dd11  kudu::tablet::TransactionTracker::~TransactionTracker() at ??:0
 4840     @     0x7f379485a60b  kudu::tablet::TabletPeer::~TabletPeer() at ??:0
 4841     @     0x7f379485a7ba  kudu::tablet::TabletPeer::~TabletPeer() at ??:0
 4842     @     0x7f3795da1998  kudu::RefCountedThreadSafe<>::DeleteInternal() at ??:0
 4843     @     0x7f3795da195a  kudu::DefaultRefCountedThreadSafeTraits<>::Destruct() at ??:0
 4844     @     0x7f3795da192a  kudu::RefCountedThreadSafe<>::Release() at ??:0
 4845     @     0x7f3795d9ed4b  scoped_refptr<>::~scoped_refptr() at ??:0
 4846     @     0x7f3795de7494  kudu::tserver::TSTabletManager::DeleteTablet() at ??:0
 4847     @     0x7f3795dc4497  kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0
 4848     @     0x7f3793982ef5  kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_1::operator()() at ??:0
 4849     @     0x7f3793982d0f  std::_Function_handler<>::_M_invoke() at ??:0
 4850     @     0x7f37934b3d49  std::function<>::operator()() at ??:0
 4851     @     0x7f37934b3949  kudu::rpc::GeneratedServiceIf::Handle() at ??:0
 4852     @     0x7f37934b569e  kudu::rpc::ServicePool::RunThread() at ??:0
 4853     @     0x7f37934b7477  boost::_mfi::mf0<>::operator()() at ??:0                                                                                                                                                                                                                                                                                                     4854     @     0x7f37934b73db  boost::_bi::list1<>::operator()<>() at ??:0
 4855     @     0x7f37934b7384  boost::_bi::bind_t<>::operator()() at ??:0                                                                                                                                                                                                                                                                                                   4856     @     0x7f37934b71aa  boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0
 4857     @     0x7f379345ee72  boost::function0<>::operator()() at ??:0
 4858     @     0x7f378ef4d560  kudu::Thread::SuperviseThread() at ??:0                                                                                                                                                                                                                                                                                                      4859     @           0x42182a  __tsan_thread_start_func at ??:0
 4860     @     0x7f379098c182  start_thread at ??:0                                                                                                                                                                                                                                                                                                                         4861     @     0x7f378c51b47d  clone at ??:0
 4862     @              (nil)  (unknown)

Issue Links

• duplicates: KUDU-1294 CHECK failure on TransactionTracker memtracker with unreleased consumption