[SPARK-48210] Modify the description of whether dynamic partitioning is enabled in the “ Stage Level Scheduling Overview”

• Type: Documentation
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Documentation
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: guihuawen
• Created: 09/May/24 03:50
• Updated: 20/May/24 03:52

Description

“  Stage Level Scheduling Overview ” in running-on-yarn and  running-on-kubernetes

The description of dynamic partitioning is inconsistent with the code implementation verification.

In running-on-yarn 

'

• When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup.

'

But the  implementation is：

Class：ResourceProfileManager

Fuc：isSupported

private[spark] def isSupported(rp: ResourceProfile): Boolean = {
assert(master != null)
if (rp.isInstanceOf[TaskResourceProfile] && !dynamicEnabled) {
if ((notRunningUnitTests || testExceptionThrown) &&
!(isStandaloneOrLocalCluster || isYarn || isK8s))

{ throw new SparkException("TaskResourceProfiles are only supported for Standalone, " + "Yarn and Kubernetes cluster for now when dynamic allocation is disabled.") }

}

 

The judgment of this code is that it does not support TaskResourceProfile in Yarn and k8s when dynamic partitioning is closed.

 

The description in the document does not match, so the document needs to be modified.

 

 

 

 

 

Issue Links

• links to: GitHub Pull Request #46496