[SPARK-20628] Keep track of nodes which are going to be shut down & avoid scheduling new tasks

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 07/May/17 03:34
• Updated: 19/Jun/20 00:49
• Resolved: 14/Feb/20 20:49

Description

Keep track of nodes which are going to be shut down. We considered adding this for YARN but took a different approach, for instances where we can't control instance termination though (EC2, GCE, etc.) this may make more sense.

Issue Links

• causes: SPARK-30981 Fix flaky "Test basic decommissioning" test
• causes: SPARK-31011 Failed to register signal handler for PWR
• duplicates: SPARK-19941 Spark should not schedule tasks on executors on decommissioning YARN nodes
• is duplicated by: SPARK-19941 Spark should not schedule tasks on executors on decommissioning YARN nodes
• is related to: SPARK-28022 k8s pod affinity to achieve cloud native friendly autoscaling
• is related to: SPARK-30610 spark worker graceful shutdown
• links to: [Github] Pull Request #19045 (holdenk)
• links to: [Github] Pull Request #19267 (juanrh)
• links to: GitHub Pull Request #19045
• links to: GitHub Pull Request #19267
• links to: GitHub Pull Request #26440
• links to: GitHub Pull Request #27588