[SPARK-3795] Add scheduler hooks/heuristics for adding and removing executors

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Andrew Or
• Reporter: Patrick Wendell
• Created: 04/Oct/14 19:06
• Updated: 30/Oct/14 00:49
• Resolved: 30/Oct/14 00:49

Description

To support dynamic scaling of a Spark application, Spark's scheduler will need to have hooks around explicitly decommissioning executors. We'll also need basic heuristics governing when to start/stop executors based on load. An initial goal is to keep this very simple.

Issue Links

• links to: [Github] Pull Request #2746 (andrewor14)