[SPARK-16630] Blacklist a node if executors won't launch on it.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.2
• Fix Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Attila Zsolt Piros
• Reporter: Thomas Graves
• Created: 19/Jul/16 19:48
• Updated: 30/Oct/20 18:29
• Resolved: 21/Jun/18 14:18

Description

On YARN, its possible that a node is messed or misconfigured such that a container won't launch on it. For instance if the Spark external shuffle handler didn't get loaded on it , maybe its just some other hardware issue or hadoop configuration issue.

It would be nice we could recognize this happening and stop trying to launch executors on it since that could end up causing us to hit our max number of executor failures and then kill the job.

Issue Links

• causes: SPARK-29683 Job failed due to executor failures all available nodes are blacklisted
• relates to: SPARK-19755 Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.
• relates to: SPARK-24567 nodeBlacklist does not get updated if a spark executor fails to launch on a mesos node
• relates to: SPARK-23485 Kubernetes should support node blacklist
• links to: [Github] Pull Request #21068 (attilapiros)