Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8379

Improve balancing resources in already satisfied queues by using Capacity Scheduler preemption

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Existing capacity scheduler only supports preemption for an underutilized queue to reach its guaranteed resource. In addition to that, there’s an requirement to get better balance between queues when all of them reach guaranteed resource but with different fairness resource.

      An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing scheduler preemption won't happen. But this is unfair to queue_a since queue_a has the same guaranteed resources.

      Before YARN-5864, capacity scheduler do additional preemption to balance queues. We changed the logic since it could preempt too many containers between queues when all queues are satisfied.

        Attachments

        1. ericpayne.confs.tgz
          5 kB
          Eric Payne
        2. YARN-8379.001.patch
          24 kB
          Zian Chen
        3. YARN-8379.002.patch
          24 kB
          Zian Chen
        4. YARN-8379.003.patch
          40 kB
          Zian Chen
        5. YARN-8379.004.patch
          50 kB
          Zian Chen
        6. YARN-8379.005.patch
          56 kB
          Zian Chen
        7. YARN-8379.006.patch
          60 kB
          Zian Chen

          Activity

            People

            • Assignee:
              Zian Chen Zian Chen
              Reporter:
              leftnoteasy Wangda Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: