[IMPALA-5598] ExecQueryFInstances RPC recv side timeouts (observed in stress test in insecure+release build)

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0, Impala 2.10.0
• Fix Version/s: Impala 2.10.0
• Component/s: Distributed Exec
• Labels: regression stress
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-2
• Assignee: Joe McDonnell
• Reporter: Michael Brown
• Created: 28/Jun/17 21:23
• Updated: 16/Aug/17 23:27
• Resolved: 16/Aug/17 23:27

Description

A run of the stress test in a 2.9-based release build has found a high number of queries that fail with one of these types of messages:

Query failed: Sender timed out waiting for receiver fragment instance: db4a82d9ad43fa4b:b8ced000000014

Query failed: ExecQueryFInstances rpc query_id=414ae4b1fade911a:fddb792300000000 failed: RPC recv timed out: Client hostname:22000 timed-out during recv call.

Roughly 400 in 10,000 which feels very high. I don't recall seeing this in 2.8-based builds. I can check if asked. Still gathering logs.

Issue Links

• is broken by: IMPALA-2550 Switch to per-query exec rpc
• relates to: IMPALA-5275 Avoid printing Status stack trace on hot paths