[SPARK-23485] Kubernetes should support node blacklist

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0, 3.0.0
• Component/s: Kubernetes, Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Imran Rashid
• Created: 21/Feb/18 22:40
• Updated: 28/Jun/22 10:26

Description

Spark's BlacklistTracker maintains a list of "bad nodes" which it will not use for running tasks (eg., because of bad hardware). When running in yarn, this blacklist is used to avoid ever allocating resources on blacklisted nodes: https://github.com/apache/spark/blob/e836c27ce011ca9aef822bef6320b4a7059ec343/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala#L128

I'm just beginning to poke around the kubernetes code, so apologies if this is incorrect – but I didn't see any references to scheduler.nodeBlacklist() in KubernetesClusterSchedulerBackend so it seems this is missing. Thought of this while looking at SPARK-19755, a similar issue on mesos.

Issue Links

• is related to: SPARK-19755 Blacklist is always active for MesosCoarseGrainedSchedulerBackend. As result - scheduler cannot create an executor after some time.
• is related to: SPARK-16630 Blacklist a node if executors won't launch on it.