[KUDU-966] [ycsb cluster] Strange multi-second pauses doing master lookups

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Feature Complete
• Fix Version/s: 0.7.0
• Component/s: master, ops-tooling
• Target Version/s: 0.7.0
• Code Review: http://gerrit.cloudera.org:8080/2053
• Assignee: Todd Lipcon
• Reporter: Jean-Daniel Cryans
• Created: 10/Aug/15 14:33
• Updated: 05/Feb/16 00:47
• Resolved: 05/Feb/16 00:47

Description

We're seeing 5 seconds pauses on a1216's master, the client blocks because it goes into a never-ending loop of waiting to long for calls to finish, while the master is taking too long servicing them.

The strange thing is that it doesn't always happen.

Here's such a call:

W0807 04:42:04.050082 10653 inbound_call.cc:189] Call kudu.master.MasterService.GetMasterRegistration from 10.20.188.113:33269 (request call id 10) took 9927ms (client timeout 10000).
W0807 04:42:04.050175 10652 inbound_call.cc:193] Trace:
0807 04:41:54.123138 (+     0us) service_pool.cc:97] Inserting onto call queue
0807 04:41:59.044863 (+4921725us) service_pool.cc:158] Handling call
0807 04:42:04.049864 (+5005001us) inbound_call.cc:110] Queueing success response