[KUDU-2011] Request-side sidecars cannot be safely destroyed on timeout

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Component/s: rpc
• Assignee: Michael Ho
• Reporter: Henry Robinson
• Created: 15/May/17 18:40
• Updated: 05/Jan/21 07:14

Description

If a timeout occurs while sending a request-side sidecar (see KUDU-1866), the RPC callback may be invoked before the outbound transfer has been completely written.

This is the last notification from the RPC layer that the caller will get, so you might expect them to delete the sidecar payload at that point, but it's not safe to do so. In fact, with a slow sender there is no way for the caller to know when it's safe to delete the payload. There's no problem for the protobuf message data, as it's serialized during the blocking part of an async call, and that memory is tied to the lifetime of the outbound call, which is managed by the RPC layer.

Ownership of the sidecar payloads should be shared between caller and the RPC layer, so really it's the new RpcSidecar::FromSlice API that causes the problems because ownership is not shared with the RpcSidecar which does have the correct lifetime. I propose removing FromSlice and having a FromFaststring(shared_ptr<faststring>) variant.

Issue Links

• is depended upon by: IMPALA-3380 Add TCP timeouts to all RPCs that don't block
• is related to: IMPALA-2567 KRPC milestone 1
• is related to: KUDU-2065 Support cancellation for outbound client RPC
• relates to: KUDU-2192 KRPC should have a timer to close stuck connections