Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4280

CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.1, 2.8.0, 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: capacity scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Consider the following scenario:

      There are 2 queues A(25% of the total capacity) and B(75%), both can run at total cluster capacity. There are 2 applications, appX that runs on Queue A, always asking for 1G containers(non-AM) and appY runs on Queue B asking for 2 GB containers.
      The user limit is high enough for the application to reach 100% of the cluster resource.

      appX is running at total cluster capacity, full with 1G containers releasing only one container at a time. appY comes in with a request of 2GB container but only 1 GB is free. Ideally, since appY is in the underserved queue, it has higher priority and should reserve for its 2 GB request. Since this request puts the alloc+reserve above total capacity of the cluster, reservation is not made. appX comes in with a 1GB request and since 1GB is still available, the request is allocated.

      This can continue indefinitely causing priority inversion.

        Attachments

        1. YARN-4280-branch-2.8.005.patch
          29 kB
          Kuhu Shukla
        2. YARN-4280-branch-2.8.004.patch
          29 kB
          Kuhu Shukla
        3. YARN-4280.014.patch
          30 kB
          Kuhu Shukla
        4. YARN-4280.013.patch
          30 kB
          Kuhu Shukla
        5. YARN-4280.012.patch
          30 kB
          Kuhu Shukla
        6. YARN-4280.011.patch
          30 kB
          Kuhu Shukla
        7. YARN-4280.010.patch
          30 kB
          Kuhu Shukla
        8. YARN-4280-branch-2.8.003.patch
          30 kB
          Kuhu Shukla
        9. YARN-4280-branch-2.8.002.patch
          30 kB
          Kuhu Shukla
        10. YARN-4280-branch-2.009.patch
          30 kB
          Wangda Tan
        11. YARN-4280.009.patch
          30 kB
          Kuhu Shukla
        12. YARN-4280.008_.patch
          30 kB
          Wangda Tan
        13. YARN-4280-branch-2.8.001.patch
          30 kB
          Kuhu Shukla
        14. YARN-4280.008.patch
          30 kB
          Kuhu Shukla
        15. YARN-4280.007.patch
          30 kB
          Kuhu Shukla
        16. YARN-4280.006.patch
          30 kB
          Kuhu Shukla
        17. YARN-4280.005.patch
          30 kB
          Kuhu Shukla
        18. YARN-4280.004.patch
          30 kB
          Kuhu Shukla
        19. YARN-4280.003.patch
          30 kB
          Kuhu Shukla
        20. YARN-4280.002.patch
          34 kB
          Kuhu Shukla
        21. YARN-4280.001.patch
          33 kB
          Kuhu Shukla

          Activity

            People

            • Assignee:
              kshukla Kuhu Shukla
              Reporter:
              kshukla Kuhu Shukla
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: