[IMPALA-8143] Add features to DoRpcWithRetry()

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Epic Color: ghx-label-3
• Assignee: Andrew Sherman
• Reporter: Andrew Sherman
• Created: 30/Jan/19 00:49
• Updated: 23/Dec/20 18:04
• Resolved: 13/Jun/19 16:20

Description

DoRpcWithRetry() is a templated utility function that is currently in control-service.h that is used to retry synchronous Krpc calls. It makes a call to a Krpc function that is is passed as a lambda function. It sets the krpc timeout to the ‘krpc_timeout‘ parameter and calls the Krpc function a number of times controlled by the ‘times_to_try’ parameter.

Possible improvements:

• Move code to rpc-mgr.inline.h
• Add a configurable sleep if RpcMgr::IsServerTooBusy() says the remote server’s queue is full.
• Do we want exponential backoff? How can we test this?
• how long should the sleep be for existing clients?
• Make QueryState::ReportExecStatus() use DoRpcWithRetry()
• Consider if asynchronous code like that in KrpcDataStreamSender::Channel  can also use DoRpcWithRetry()
• Replace FAULT_INJECTION_RPC_DELAY with DebugAction 

Issue Links

• is related to: IMPALA-7283 Consider backing off before retrying RPC