[SPARK-39853] Support stage level schedule for standalone cluster when dynamic allocation is disabled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Tengfei Huang
• Reporter: Tengfei Huang
• Created: 24/Jul/22 15:23
• Updated: 27/Feb/24 14:52
• Resolved: 30/Sep/22 01:06

Description

SPARK-39062 added stage level schedule support for standalone cluster when dynamic allocation was enabled, spark would request for executors for different resource profiles.
While when dynamic allocation is disabled, we can also leverage stage level schedule to schedule tasks based on resource profile(task resource requests) to executors with default resource profile.

Issue Links

• breaks: SPARK-45527 Task fraction resource request is not expected
• causes: SPARK-41848 Tasks are over-scheduled with TaskResourceProfile
• links to: [Github] Pull Request #37268 (ivoson)