[SPARK-29149] Update YARN cluster manager For Stage Level Scheduling

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core, YARN
• Epic Link: Stage Level Scheduling
• Assignee: Thomas Graves
• Reporter: Thomas Graves
• Created: 18/Sep/19 12:33
• Updated: 17/May/20 18:14
• Resolved: 28/Feb/20 21:25

Description

For Stage Level Scheduling, we need to update the YARN allocator to handle requesting executors for multiple ResourceProfiles.

• The container requests have to be updated to be based on a number of containers per ResourceProfile.  This is a larger change then you might expect because on YARN you can’t ask for different container sizes within the same YARN container priority.  So we will have to ask for containers at different priorities to be able to get different container sizes. Other YARN applications like Tez handle this now so it shouldn’t be a big deal just a matter of mapping stages to different priorities.
• The allocation response from YARN has to match the containers to a resource profile.
• We need to launch the container with additional parameters so the executor knows its resource profile to report back to the ExecutorMonitor.  

 

Issue Links

• links to: GitHub Pull Request #27583