[SPARK-30750] stage level scheduling: Add ability to set dynamic allocation configs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Epic Link: Stage Level Scheduling
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 06/Feb/20 15:50
• Updated: 06/Feb/20 15:50

Description

the initial Jira to modify dynamic allocation for stage level scheduling applies the configs - minimum, initial, max number of executors to each profile individually. This means that you can't set those different per resource profile.

Ideally those would be configurable per ResourceProfile so look at adding support for that.