[YARN-8820] [Umbrella] GPU support on YARN - Phase 2

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: yarn
• Assignee: Unassigned
• Reporter: Zhankun Tang
• Created: 25/Sep/18 08:41
• Updated: 26/Nov/18 13:17

Description

In YARN-6223, we've done a basic support for Nvidia GPU on YARN including resource discovery, allocation, cgroups isolation as well as docker support (Nvidia-docker v1). But there's still room for us to improve.

For instance, multiple GPU cards in one host bring the requirements of GPU hierarchy scheduling. The Nvidia-docker v2 emerged and v1 has been deprecated. And we're planning a new device plugin framework in YARN which has relation to GPU support too. (maybe in the long term)

So here we converge threads related to the above and open an umbrella here to track the next stage tasks for convenience.

One thing to note is that a pluggable device framework is in progress (YARN-8851), once that framework is mature, we should prefer to utilize the ability of the framework to achieve these phase 2 support.

Sub-Tasks

• [YARN-8851] GPU hierarchy/topology scheduling support based on pluggable device framework: Resolved
• Nvidia-docker v2 support for YARN GPU feature: Resolved
• Monitor the healthy state of GPU: Open