[SPARK-29154] Update Spark scheduler for stage level scheduling

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Scheduler, Spark Core
• Epic Link: Stage Level Scheduling
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 18/Sep/19 15:35
• Updated: 24/Nov/20 20:56
• Resolved: 26/Mar/20 14:49

Description

Make the changes to DAGscheduler, stage, task set manager, task scheduler to support scheduling based on the resource profiles.  Note that the logic to merge profiles has a separate jira.

Issue Links

• links to: GitHub Pull Request #27773
• links to: GitHub Pull Request #28108