[SPARK-45495] Support stage level task resource profile for k8s cluster when dynamic allocation disabled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Bobby Wang
• Reporter: Bobby Wang
• Created: 11/Oct/23 00:38
• Updated: 13/Oct/23 15:52
• Resolved: 13/Oct/23 15:52

Description

https://github.com/apache/spark/pull/37268 has introduced a new feature that supports stage-level schedule task resource profile for standalone cluster when dynamic allocation is disabled. It's really cool feature, especially for ML/DL cases, more details can be found in that PR.

 

The problem here is that the feature is only available for standalone and YARN cluster for now, but most users would also expect it can be used for other spark clusters like K8s.

 

So I filed this issue to track this task.

Issue Links

• is a clone of: SPARK-45250 Support stage level task resource profile for yarn cluster when dynamic allocation disabled
• links to: GitHub Pull Request #43323