[KUDU-1052] TSTabletManager startup very slow with lots of orphaned blocks

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Private Beta
• Fix Version/s: n/a
• Component/s: tserver
• Target Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 21/Aug/15 15:39
• Updated: 16/May/16 00:03
• Resolved: 16/May/16 00:03

Description

I'm restarting a TS with a couple hundred tablets on it. The TS had recently fallen on bad times, and had a bunch of tablets which probably were somehow in the process of deletion, etc. The TS is now taking a long time to start up in stacks like:
Thread 1 (Thread 0x7fa8292c6860 (LWP 5893)):
#0 0x00000032e52e189d in fdatasync () from /lib64/libc.so.6
#1 0x00000000012352c5 in ?? ()
#2 0x0000000001265054 in kudu::pb_util::WritablePBContainerFile::Sync() ()
#3 0x00000000011fbdd8 in kudu::fs::LogBlockManager::DeleteBlock(kudu::BlockId const&) ()
#4 0x00000000011f25f8 in kudu::FsManager::DeleteBlock(kudu::BlockId const&) ()
#5 0x000000000085cd2d in kudu::tablet::TabletMetadata::DeleteOrphanedBlocks(std::vector<kudu::BlockId, std::allocator<kudu::BlockId> > const&) ()
#6 0x000000000085ebff in kudu::tablet::TabletMetadata::LoadFromSuperBlock(kudu::tablet::TabletSuperBlockPB const&) ()
#7 0x000000000085ee2e in kudu::tablet::TabletMetadata::LoadFromDisk() ()
#8 0x000000000085f006 in kudu::tablet::TabletMetadata::Load(kudu::FsManager*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<kudu::tablet::TabletMetadata>*) ()
#9 0x00000000006f584e in kudu::tserver::TSTabletManager::OpenTabletMeta(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, scoped_refptr<kudu::tablet::TabletMetadata>*) ()
#10 0x00000000006fc246 in kudu::tserver::TSTabletManager::Init() ()
#11 0x00000000006e3caf in kudu::tserver::TabletServer::Init() ()

Issue Links

• relates to: KUDU-1061 DeleteTablet should asynchronously (or batch) delete blocks