[SPARK-29148] Modify dynamic allocation manager for stage level scheduling

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Scheduler, Spark Core
• Epic Link: Stage Level Scheduling
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 18/Sep/19 12:29
• Updated: 17/Jun/20 02:48
• Resolved: 12/Feb/20 22:46

Description

To Support Stage Level Scheduling, the dynamic allocation manager has to track the usage and need or executor per ResourceProfile.

We will have to figure out what to do with the metrics.

Issue Links

• is related to: SPARK-30942 Fix the warning for requiring cores to be limiting resource
• links to: [Github] Pull Request #28847 (dongjoon-hyun)
• links to: [Github] Pull Request #28847 (dongjoon-hyun)
• links to: GitHub Pull Request #27313
• links to: GitHub Pull Request #27615