[KUDU-383] Massive numbers of threads used by log append and GC

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: M4.5
• Fix Version/s: n/a
• Component/s: log
• Labels: data-scalability
• Target Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Mike Percy
• Created: 01/Jul/14 23:26
• Updated: 13/Apr/17 02:42
• Resolved: 13/Apr/17 02:33

Description

$ ../../build-support/stacktrace-thread-summary.pl create-table-stress-test-10697.txt | awk '{print $3}' | sort | uniq -c | sort -n
      1 kudu::KernelStackWatchdog::RunThread()
      1 kudu::MaintenanceManager::RunSchedulerThread()
      1 kudu::master::CatalogManagerBgTasks::Run()
      1 kudu::tserver::Heartbeater::Thread::RunThread()
      1 kudu::tserver::ScannerManager::RunRemovalThread()
      1 main
      1 timer_helper_thread
      1 timer_sigev_thread
      2 kudu::rpc::AcceptorPool::RunThread()
      2 master_thread
      4 kudu::ThreadPool::DispatchThread(bool)
     12 kudu::rpc::ReactorThread::RunThread()
     20 kudu::rpc::ServicePool::RunThread()
   3291 kudu::log::Log::AppendThread::RunThread()
   3291 kudu::tablet::TabletPeer::RunLogGC()

W0626 02:09:16.853266 27997 messenger.cc:187] Unable to handle RPC call: Service unavailable: TSHeartbeat request on kudu.master.MasterService from 127.0.0.1:46937 dropped due to backpressure. The service queue is full; it has 50 items.
W0626 02:09:16.854862 28074 heartbeater.cc:278] Failed to heartbeat: Remote error: Failed to send heartbeat: Service unavailable: TSHeartbeat request on kudu.master.MasterService from 127.0.0.1:46937 dropped due to backpressure. The service queue is full; it has 50 items.
W0626 02:09:16.882686 27997 messenger.cc:187] Unable to handle RPC call: Service unavailable: TSHeartbeat request on kudu.master.MasterService from 127.0.0.1:46937 dropped due to backpressure. The service queue is full; it has 50 items.
W0626 02:09:16.884294 28074 heartbeater.cc:278] Failed to heartbeat: Remote error: Failed to send heartbeat: Service unavailable: TSHeartbeat request on kudu.master.MasterService from 127.0.0.1:46937 dropped due to backpressure. The service queue is full; it has 50 items.
F0626 02:09:31.747577 10965 test_main.cc:63] Maximum unit test time exceeded (900 sec)

Issue Links

• duplicates: KUDU-1913 Tablet server runs out of threads when creating lots of tablets