[SPARK-31437] Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Hongze Zhang
• Created: 13/Apr/20 12:41
• Updated: 20/May/20 14:06

Description

By the change in PR of SPARK-29154, submitted tasks are scheduled onto executors only if resource profile IDs strictly match. As a result Spark always starts new executors for customized ResourceProfiles.

This limitation makes working with process-local jobs unfriendly. E.g. Task cores has been increased from 1 to 4 in a new stage, and executor has 8 slots, it is expected that 2 new tasks can be run on the existing executor but Spark starts new executors for new ResourceProfile. The behavior is unnecessary.

Issue Links

• relates to: SPARK-27495 SPIP: Support Stage level resource configuration and scheduling