[YARN-6223] [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 23/Feb/17 00:46
• Updated: 14/Nov/18 17:38
• Resolved: 06/Apr/18 18:32

Description

As varieties of workloads are moving to YARN, including machine learning / deep learning which can speed up by leveraging GPU computation power. Workloads should be able to request GPU from YARN as simple as CPU and memory.

To make a complete GPU story, we should support following pieces:
1) GPU discovery/configuration: Admin can either config GPU resources and architectures on each node, or more advanced, NodeManager can automatically discover GPU resources and architectures and report to ResourceManager

2) GPU scheduling: YARN scheduler should account GPU as a resource type just like CPU and memory.

3) GPU isolation/monitoring: once launch a task with GPU resources, NodeManager should properly isolate and monitor task's resource usage.

For #2, YARN-3926 can support it natively. For #3, YARN-3611 has introduced an extensible framework to support isolation for different resource types and different runtimes.

Related JIRAs:
There're a couple of JIRAs (YARN-4122/YARN-5517) filed with similar goals but different solutions:
For scheduling:

• YARN-4122/YARN-5517 are all adding a new GPU resource type to Resource protocol instead of leveraging YARN-3926.

For isolation:

• And YARN-4122 proposed to use CGroups to do isolation which cannot solve the problem listed at https://github.com/NVIDIA/nvidia-docker/wiki/GPU-isolation#challenges such as minor device number mapping; load nvidia_uvm module; mismatch of CUDA/driver versions, etc.

Issue Links

• is related to: YARN-8200 Backport resource types/GPU features to branch-3.0/branch-2
• relates to: YARN-4122 Add support for GPU as a resource
• relates to: YARN-5983 [Umbrella] Support for FPGA as a Resource in YARN
• requires: YARN-3926 [Umbrella] Extend the YARN resource model for easier resource-type management and profiles

Sub-Tasks

• Add support for GPU as a resource: Resolved
• Add support in NodeManager to isolate GPU devices by using CGroups: Resolved
• [YARN-6223] Native code changes to support isolate GPU devices by using CGroups: Resolved
• Document GPU isolation feature: Resolved
• Support GPU isolation for docker container: Resolved
• Add support to show GPU in UI including metrics: Resolved
• GPU Isolation: Incorrect minor device numbers written to devices.deny file: Resolved
• Use "docker volume inspect" to make sure that volumes for GPU drivers/libs are properly mounted.: Resolved
• Ensure volume to include GPU base libraries after created by plugin: Resolved
• Gpu Information page could be empty for nodes without GPU: Resolved
• GPU volume creation command fails when work preserving is disabled at NM: Resolved
• Document YARN Ambari Integration Guide for GPU: Resolved