[IMPALA-7449] TotalNetworkThroughput in KrpcDataStreamSender is broken

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 3.1.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-1
• Assignee: Michael Ho
• Reporter: Michael Ho
• Created: 15/Aug/18 05:58
• Updated: 21/Aug/18 21:42
• Resolved: 21/Aug/18 21:42

Description

The network throughput computation fails to take into account of the fact that multiple RPCs can be happening in parallel. Currently, the throughput is computed by (total bytes sent / total network time). The total network time is the aggregate of the network time observed of each RPC. This seems hard to understand (or wrong?) when there are drastically different throughput when sending to different hosts. It may be slightly easier to understand if we switch to measuring the observed network throughput of each individual RPC and use a summary counter or a histogram to record the throughput.

Issue Links

• is broken by: IMPALA-6685 Improve profile in KrpcDataStreamRecvr and KrpcDataStreamSender