Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5517

Add GPU as a resource type for scheduling

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None

      Description

      Currently YARN only support scheduling based on memory and cpu.
      There is the issue(YARN-3926) which proposed to extend the YARN resource model.
      And there is the issue(YARN-4122) to add support for GPU as a resource using docker.

      But these issues didn’t release yet so I just added GPU resource type like memory and cpu.
      I don’t consider GPU isolation like YARN-4122.

      The properties for GPU resource type is similar to cpu core.

      mapred-default.xml
      mapreduce.map.gpu.cores (default 0)
      mapreduce.reduce.gpu.cores (default 0)
      yarn.app.mapreduce.am.resource.gpu-cores (default 0)

      yarn-default.xml
      yarn.scheduler.minimum-allocation-gcores (default 0)
      yarn.scheduler.maximum-allocation-gcores (default 8)
      yarn.nodemanager.resource.gcores (default 0)

      I attached the patch for branch-2.7.1

        Attachments

        1. RM-scheduler_metrics.jpg
          37 kB
          Jaeboo Jeong
        2. aggregate_resource_allocation.jpg
          79 kB
          Jaeboo Jeong
        3. container_example.jpg
          49 kB
          Jaeboo Jeong
        4. YARN-5517-branch-2.7.1.patch
          556 kB
          Jaeboo Jeong

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jaeboo Jaeboo Jeong
              • Votes:
                0 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: