[SPARK-20327] Add CLI support for YARN custom resources, like GPUs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 3.0.0
• Component/s: Spark Shell, Spark Submit
• Labels: newbie
• Assignee: Szilard Nemeth
• Reporter: Daniel Templeton
• Created: 13/Apr/17 16:41
• Updated: 17/Oct/18 15:42
• Resolved: 13/Oct/18 01:15

Description

YARN-3926 adds the ability for administrators to configure custom resources, like GPUs. This JIRA is to add support to Spark for requesting resources other than CPU virtual cores and memory. See YARN-3926.

Issue Links

• Dependent: SPARK-23151 Provide a distribution of Spark with Hadoop 3.0
• depends upon: YARN-3926 [Umbrella] Extend the YARN resource model for easier resource-type management and profiles
• links to: [Github] Pull Request #20761 (szyszy)
• links to: [Github] Pull Request #22751 (vanzin)
• links to: [Github] Pull Request #22751 (vanzin)