[KUDU-2395] Thread spike with all threads blocked in libnss

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: consensus, tserver, util
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 30/Mar/18 23:57
• Updated: 03/Jun/20 02:41
• Resolved: 03/Jun/20 02:41

Description

I saw the thread count on a server under a load test spike from 280 threads (fairly constant) to 3400 threads (briefly). I checked the diagnostics log and found that there are several thousand threads in a stack like:

0x7facce018606 _nss_files_gethostbyname2_r
  0x345a703645 <unknown>
  0x345a6d0b3b <unknown>
  0x345a6d2d80 <unknown>
     0x1c9366c kudu::(anonymous namespace)::GetAddrInfo()
     0x1c95fbe kudu::HostPort::ResolveAddresses()
      0xac4b78 kudu::consensus::(anonymous namespace)::CreateConsensusServiceProxyForHost()
      0xac5058 kudu::consensus::RpcPeerProxyFactory::NewProxy()
      0xb0b212 kudu::consensus::LeaderElection::LeaderElection()
      0xafab80 kudu::consensus::RaftConsensus::StartElection()
      0xafd20c kudu::consensus::RaftConsensus::ReportFailureDetectedTask()
     0x1ccf4ed kudu::FunctionRunnable::Run()

Issue Links

• is related to: KUDU-2244 spinlock contention in raft_consensus
• relates to: KUDU-2791 Consider adding a short ttl-cache for dns lookups
• relates to: KUDU-75 Allow RPC proxies to take HostPort and do DNS resolution inline with calls