[SPARK-31378] stage level scheduling dynamic allocation bug with initial num executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 07/Apr/20 14:38
• Updated: 07/Apr/20 21:56
• Resolved: 07/Apr/20 21:56

Description

I found a bug in the stage level scheduling dynamic allocation code when you have a non default profile and it has an initial number of executors the same as what the number of executors needed for the first job, then we don't properly request the executors.

Issue Links

• relates to: SPARK-27495 SPIP: Support Stage level resource configuration and scheduling
• links to: GitHub Pull Request #28146