[IMPALA-10472] Add a flag to control Kudu client connection negotiation timeout in backend

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: Kudu client rpc
• Epic Color: ghx-label-8
• Assignee: Alexey Serbin
• Reporter: Alexey Serbin
• Created: 03/Feb/21 23:51
• Updated: 03/Dec/21 05:54
• Resolved: 08/Mar/21 02:30

Description

Since KUDU-2966 is addressed, it's now possible to control the RPC connection negotiation timeout from the Kudu client's side (C++ client). To use the newly introduced functionality in Impala's back-end, it's necessary to provide a control knob for that.

This should help to address cases where busy/overloaded cluster nodes hosting Kudu tablet servers aren't fast enough to negotiate new connections within the default timeout interval (in most cases that's about Kudu server's connection negotiation threads being scheduled as needed and getting enough CPU time). In practice, it's necessary to customize the corresponding setting on the Kudu server side: it's controlled by the --rpc_negotiation_timeout_ms flag for kudu-master and kudu-tserver.

The idea is to add a gflag named like --kudu_client_connection_negotiation_timeout_ms with default value of 3000. The default value 3000 is to keep the new code backwards-compatible.