[IMPALA-6048] Queries make very slow progress and report WaitForRPC() stuck for too long

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.11.0
• Fix Version/s: Not Applicable
• Component/s: Backend, Distributed Exec
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-1
• Assignee: Michael Ho
• Reporter: Mostafa Mokhtar
• Created: 12/Oct/17 23:54
• Updated: 14/May/19 01:53
• Resolved: 14/May/19 01:53

Description

When running 32 concurrent queries from TPCDS a couple of instances from TPC-DS Q78 9 hours to finish and it appeared to be hung.

On an idle cluster the query finished in under 5 minutes, profiles attached.
When the query ran for long fragments reported +16 hours of network send/receive time

The logs show there is a lot of messages like the one below, there are incidents for this log message where a node waited too long from an RPC from itself

W1012 00:47:57.633549 117475 krpc-data-stream-sender.cc:360] XXX: WaitForRPC() stuck for too long address=10.17.234.37:29000 fragment_instace_id_=1e48ef897e797131:2f05789b000005eb dest_node_id_=24 sender_id_=81

Issue Links

• is duplicated by: IMPALA-6294 Concurrent hung with lots of spilling make slow progress due to blocking in DataStreamRecvr and DataStreamSender
• is related to: IMPALA-6294 Concurrent hung with lots of spilling make slow progress due to blocking in DataStreamRecvr and DataStreamSender