[IMPALA-8339] Coordinator should be more resilient to fragment instances startup failure

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Component/s: Distributed Exec
• Labels: Availability resilience
• Epic Color: ghx-label-7
• Assignee: Thomas Tauber-Marshall
• Reporter: Michael Ho
• Created: 22/Mar/19 22:23
• Updated: 16/Jan/20 21:04
• Resolved: 30/Jul/19 15:22

Description

Impala currently relies on statestore for cluster membership. When an Impala executor goes offline, it may take a while for statestore to declare that node as unavailable and for that information to be propagated to all coordinator nodes. Within this window, some coordinator nodes may still attempt to issue RPCs to the faulty node, resulting in RPC failures which resulted in query failures. In other words, many queries may fail to start within this window until all coordinator nodes get the latest information on cluster membership.

Going forward, coordinator may need to fall back to using backup executors for each fragments in case some of the executors are not available. Moreover, coordinator should treat the cluster membership information from statestore (or any external source of truth e.g. etcd) as hints instead of ground truth and adjust the scheduling of fragment instances based on the availability of the executors from the coordinator's perspective.

Issue Links

• blocks: IMPALA-2638 Retry queries that fail during scheduling
• relates to: IMPALA-9224 Blacklist nodes with faulty disks
• relates to: IMPALA-9137 Blacklist node if a DataStreamService RPC to the node fails
• relates to: IMPALA-9243 Coordinator Web UI should list which executors have been blacklisted
• relates to: IMPALA-9124 Transparently retry queries that fail due to cluster membership changes