[SPARK-27361] YARN support for GPU-aware scheduling

• Type: Story
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Epic Link: GPU-aware Scheduling
• Assignee: Thomas Graves
• Reporter: Xiangrui Meng
• Created: 04/Apr/19 00:33
• Updated: 17/May/20 18:14
• Resolved: 30/May/19 21:45

Description

Design and implement YARN support for GPU-aware scheduling:

• User can request GPU resources at Spark application level.
• How the Spark executor discovers GPU's when run on YARN
• Integrate with YARN 3.2 GPU support.

Issue Links

• blocks: SPARK-27760 Spark resources - user configs change .count to be .amount
• is blocked by: SPARK-27024 Executor interface for cluster managers to support GPU resources
• is blocked by: SPARK-23710 Upgrade the built-in Hive to 2.3.5 for hadoop-3.2
• relates to: SPARK-27959 Change YARN resource configs to use .amount
• links to: GitHub Pull Request #24634

Sub-Tasks

• Design: YARN supports Spark GPU-aware scheduling: Resolved
• Upgrade YARN to 3.1.2+ to support GPU: Resolved
• spark-submit requests GPUs in YARN mode: Resolved
• YARN passes GPU info to Spark executor: Resolved