[IMPALA-5441] Send larger row batches over the wire

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.10.0
• Component/s: Perf Investigation
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Henry Robinson
• Created: 06/Jun/17 00:06
• Updated: 21/Dec/20 19:08

Description

Our on-the-wire row batch size is the same as the in-memory size (1024 rows by default). It might make sense to increase the wire-size to reduce the RPC-per-row overhead, and decrease context-switching in the receiver.

KRPC makes it quite natural to do that: each row batch can be serialized as a sidecar in memory-size batches. The receiver can then read each batch in turn as though it were sent individually, without any need to stitch together (or split up) serialized batches.

Issue Links

• depends upon: IMPALA-2567 KRPC milestone 1