[KUDU-1409] Make krpc call timeouts more resistant to process pauses

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: rpc
• Target Version/s: 0.9.0
• Code Review: http://gerrit.cloudera.org:8080/#/c/2745/
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 11/Apr/16 02:48
• Updated: 30/Apr/16 20:57
• Resolved: 30/Apr/16 20:57

Description

In stress testing Impala on Kudu I've seen various RPC timeouts that turn out to be due to pauses on the client side. In particular, scenarios like https://issues.cloudera.org/browse/IMPALA-2800 can cause the memory allocator inside Impala to block for several seconds, and that might cause us to think we missed a timeout.

We should be more resilient to this sort of "false" timeout.