[SPARK-30749] stage level scheduling: Better cleanup of Resource profiles

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Epic Link: Stage Level Scheduling
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 06/Feb/20 15:48
• Updated: 06/Feb/20 15:48

Description

In the initial stage level scheduling jiras we aren't doing a great job of cleaning up datastructures when the ResourceProfile is done being used. This is mostly because its hard to tell when its done being used.

We should find a way to clean up better.

Also in the dynamic allocation manager, if the resource profile is not being used anymore we should be more active about getting rid of the executors that are up. Especially if there is a minimum number that is set, we can kill those off.

 

I think this can be done as followup to the main feature