[KUDU-2192] KRPC should have a timer to close stuck connections

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: rpc
• Assignee: Unassigned
• Reporter: Michael Ho
• Created: 18/Oct/17 00:52
• Updated: 27/Jul/22 02:50

Description

If the remote host goes down or its network gets unplugged, all pending RPCs to that host will be stuck if there is no timeout specified. While those RPCs which have finished sending their payloads or those which haven't started sending payloads can be cancelled quickly, those in mid-transmission (i.e. an RPC at the front of the outbound queue with part of its payload sent already) cannot be cancelled until the payload has been completely sent. Therefore, it's beneficial to have a timeout to kill a connection if it's not making any progress for an extended period of time so the RPC will fail and get unstuck. The timeout may need to be conservatively large to avoid aggressive closing of connections due to transient network issue. One can consider augmenting the existing maintenance thread logic which checks for idle connection to check for this kind of timeout. Please feel free to propose other alternatives (e.g. TPC keepalive timeout) in this JIRA.

Issue Links

• is related to: IMPALA-6159 DataStreamSender should transparently handle some connection reset by peer
• is related to: KUDU-2011 Request-side sidecars cannot be safely destroyed on timeout
• is related to: IMPALA-2567 KRPC milestone 1
• relates to: KUDU-3387 Enable TCP keepalive for connections created by Java client