[IMPALA-10141] Include aggregate TCP metrics in per-node profiles

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Distributed Exec
• Epic Link: Network Debugging / Supportability Improvements
• Epic Color: ghx-label-3
• Assignee: Unassigned
• Reporter: Sahil Takiar
• Created: 02/Sep/20 21:26
• Updated: 21/Dec/20 19:51

Description

The /rpcz endpoint in the debug web ui includes a ton of useful TCP-level metrics per kRPC connection for all inbound / outbound connections. It would be useful to aggregate some of these metrics and put them in the per-node profiles. Since it is not possible to currently split these metrics out per query, they should be added at the per-host level. Furthermore, only metrics that can be sanely aggregated across all connections should be included. For example, tracking the number of Retransmitted TCP Packets across all connections for the duration of the query would be useful. TCP retransmissions should be rare and are typically indicate of network hardware issues or network congestions, having at least some high level idea of the number of TCP retransmissions that occur during a query can drastically help determine if the network is to blame for query slowness.

Issue Links

• is related to: IMPALA-9046 Profile counter that indicates if a process or JVM pause occurred