[IMPALA-6705] TotalNetworkSendTime in query profile is misleading

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
• Component/s: Distributed Exec
• Labels: observability
• Epic Link: Network Debugging / Supportability Improvements
• Target Version: Impala 2.13.0
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Michael Ho
• Created: 19/Mar/18 23:55
• Updated: 02/Sep/20 17:44

Description

TotalNetworkSendTime is actually measuring the time which a fragment instance execution thread spent waiting for the completion of previous RPC. This is a combination of:

• network time of sending the RPC payload to the destination
• processing and queuing time in the destination
• network time of sending the RPC response to the originating node

The name of this metric itself is misleading because it gives the impression that it's the time spent sending the RPC payload to the destination so a query profile with a high TotalNetworkSendTime may easily mislead a user into concluding that there is something wrong with the network. In reality, the receiving end could be overloaded and it's taking a huge amount of time to respond to an RPC.

For this metric to be useful, we need to have a breakdown of those 3 components above.