[IMPALA-3380] Add TCP timeouts to all RPCs that don't block

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 4.0.0
• Component/s: Distributed Exec
• Labels: observability supportability
• Target Version: Product Backlog, Impala 2.13.0
• Assignee: Sahil Takiar
• Reporter: Henry Robinson
• Created: 20/Apr/16 00:24
• Updated: 14/Oct/20 19:36
• Resolved: 10/Jul/20 17:22

Description

Most RPCs should not take an unbounded amount of time to complete (the exception is TransmitData(), but that may also change). To handle hang failures on the remote machine, we should add timeouts to every RPC (so, really, every RPC client), and handle the timeout failure.

Issue Links

• depends upon: KUDU-2011 Request-side sidecars cannot be safely destroyed on timeout
• depends upon: KUDU-2065 Support cancellation for outbound client RPC
• is blocked by: IMPALA-3575 Impala should retry backend connection request and apply a send timeout
• is related to: IMPALA-2638 Retry queries that fail during scheduling
• Parent Feature: IMPALA-2990 Coordinator should timeout and cancel queries with unresponsive / stuck executors
• Parent Feature: IMPALA-5394 Set socket timeouts while opening TSaslTransport
• relates to: IMPALA-10241 Impala Doc: RPC troubleshooting guide
• relates to: IMPALA-10139 Slow RPC logs can be misleading
• relates to: IMPALA-2864 Ensure that client connections are closed after a failed RPC and a failed Open()