Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8820

[Umbrella] GPU support on YARN - Phase 2

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None

      Description

      In YARN-6223, we've done a basic support for Nvidia GPU on YARN including resource discovery, allocation, cgroups isolation as well as docker support (Nvidia-docker v1). But there's still room for us to improve.

      For instance, multiple GPU cards in one host bring the requirements of GPU hierarchy scheduling. The Nvidia-docker v2 emerged and v1 has been deprecated. And we're planning a new device plugin framework in YARN which has relation to GPU support too. (maybe in the long term)

      So here we converge threads related to the above and open an umbrella here to track the next stage tasks for convenience.

      One thing to note is that a pluggable device framework is in progress (YARN-8851), once that framework is mature, we should prefer to utilize the ability of the framework to achieve these phase 2 support.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tangzhankun Zhankun Tang
            • Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated: