We're seeing 5 seconds pauses on a1216's master, the client blocks because it goes into a never-ending loop of waiting to long for calls to finish, while the master is taking too long servicing them.
The strange thing is that it doesn't always happen.
Here's such a call:
W0807 04:42:04.050082 10653 inbound_call.cc:189] Call kudu.master.MasterService.GetMasterRegistration from 10.20.188.113:33269 (request call id 10) took 9927ms (client timeout 10000).
W0807 04:42:04.050175 10652 inbound_call.cc:193] Trace:
0807 04:41:54.123138 (+ 0us) service_pool.cc:97] Inserting onto call queue
0807 04:41:59.044863 (+4921725us) service_pool.cc:158] Handling call
0807 04:42:04.049864 (+5005001us) inbound_call.cc:110] Queueing success response