[IMPALA-9299] Node Blacklisting: Coordinators should blacklist unhealthy nodes

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Epic Color: ghx-label-10
• Assignee: Wenzhe Zhou
• Reporter: Sahil Takiar
• Created: 16/Jan/20 20:55
• Updated: 02/Apr/21 17:06

Description

Top level JIRA for Node Blacklisting.

High level description of node blacklisting, from IMPALA-8339:

This patch adds the concept of a blacklist of executors to the
coordinator, which removes executors from consideration for query
scheduling. Blacklisting decisions are local to a given coordinator
and are not included in statestore updates.

The intention is to allow coordinators to be more aggressive about
deciding that an exeutor is unhealthy or unavailable, to minimize
failed queries in environments where cluster membership may be more
variable, rather than having to wait on the statestore heartbeat
mechanism to decide that the executor is down.

Issue Links

• is related to: IMPALA-4317 Single Overloaded Impalad Causes Entire Cluster to Hang
• is related to: IMPALA-9124 Transparently retry queries that fail due to cluster membership changes
• is related to: IMPALA-10599 The coordinator should avoid distributing query fragments to busy nodes.
• relates to: IMPALA-7872 Extended health checks to mark node as down

Sub-Tasks

• Coordinator should be more resilient to fragment instances startup failure: Resolved
• Blacklist node if a DataStreamService RPC to the node fails: Resolved
• Log node blacklisting removals in ClusterMembershipMgr: Resolved
• Coordinator Web UI should list which executors have been blacklisted: Resolved
• Membership updates should only remove quiescing nodes from the blacklist: Resolved
• Impala Doc: Add docs for node blacklisting: Open
• Blacklist nodes with faulty disks: Resolved
• Failed Task Blacklisting: Resolved
• Blacklist additional posix error codes for failed DataStreamService RPCs: Open
• Add a limit on the number of nodes that can be blacklisted per query: Open
• Aux error info should detect multiple RPC failures: Open