[KUDU-1692] Deleting large tablets causes a lot of tcmalloc contention

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: tablet, util
• Code Review: http://gerrit.cloudera.org:8080/4708
• Assignee: Adar Dembo
• Reporter: Todd Lipcon
• Created: 11/Oct/16 00:09
• Updated: 20/Oct/16 18:11
• Resolved: 20/Oct/16 18:11

Description

I deleted a large table which contained about 1TB of data per tablet server. The tablet servers then started spending a large amount of time in this stack:

                  855e94 tcmalloc::ThreadCache::GetThreadStats(unsigned long*, unsigned long*) (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  84e9ba ExtractStats(TCMallocStats*, unsigned long*, tcmalloc::PageHeap::SmallSpanStats*, tcmalloc::PageHeap::LargeSpanStats*) (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-releas
                  850f8f TCMallocImplementation::GetNumericProperty(char const*, unsigned long*) (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                 1a18c50 kudu::GetTCMallocCurrentAllocatedBytes() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                 1a19a50 kudu::MemTracker::UpdateConsumption() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  980f01 std::_Sp_counted_ptr<kudu::cfile::CFileReader*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  99a937 kudu::tablet::CFileSet::~CFileSet() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  99ad61 kudu::tablet::CFileSet::~CFileSet() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  948b42 kudu::tablet::DiskRowSet::~DiskRowSet() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)
                  965f35 kudu::tablet::RowSetTree::~RowSetTree() (/opt/cloudera/parcels/KUDU-1.0.0-1.kudu1.0.0.p0.6/lib/kudu/sbin-release/kudu-tserver)