Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4148

When killing app, RM releases app's resource before they are released by NM

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When killing a app, RM scheduler releases app's resource as soon as possible, then it might allocate these resource for new requests. But NM have not released them at that time.

      The problem was found when we supported GPU as a resource(YARN-4122). Test environment: a NM had 6 GPUs, app A used all 6 GPUs, app B was requesting 3 GPUs. Killed app A, then RM released A's 6 GPUs, and allocated 3 GPUs to B. But when B tried to start container on NM, NM found it didn't have 3 GPUs to allocate because it had not released A's GPUs.

      I think the problem also exists for CPU/Memory. It might cause OOM when memory is overused.

        Attachments

        1. free_in_scheduler_but_not_node_prototype-branch-2.7.patch
          13 kB
          Jason Lowe
        2. YARN-4148.001.patch
          21 kB
          Jun Gong
        3. YARN-4148.002.patch
          21 kB
          Jason Lowe
        4. YARN-4148.003.patch
          21 kB
          Jason Lowe
        5. YARN-4148.wip.patch
          18 kB
          Jun Gong
        6. YARN-4148-branch-2.8.003.patch
          22 kB
          Jason Lowe

          Issue Links

            Activity

              People

              • Assignee:
                jlowe Jason Lowe
                Reporter:
                hex108 Jun Gong
              • Votes:
                0 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: