Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2492 (Clone of YARN-796) Allow for (admin) labels on nodes and resource-requests
  3. YARN-2933

Capacity Scheduler preemption policy should only consider capacity without labels temporarily

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.7.0
    • capacityscheduler
    • None
    • Reviewed

    Description

      Currently, we have capacity enforcement on each queue for each label in CapacityScheduler, but we don't have preemption policy to support that. YARN-2498 is targeting to support preemption respect node labels, but we have some gaps in code base, like queues/FiCaScheduler should be able to get usedResource/pendingResource, etc. by label. These items potentially need to refactor CS which we need spend some time carefully think about.

      For now, what immediately we can do is allow calculate ideal_allocation and preempt containers only for resources on nodes without labels, to avoid regression like: A cluster has some nodes with labels and some not, assume queueA isn't satisfied for resource without label, but for now, preemption policy may preempt resource from nodes with labels for queueA, that is not correct.

      Again, it is just a short-term enhancement, YARN-2498 will consider preemption respecting node-labels for Capacity Scheduler which is our final target.

      Attachments

        1. YARN-2933-1.patch
          9 kB
          Mayank Bansal
        2. YARN-2933-2.patch
          11 kB
          Mayank Bansal
        3. YARN-2933-3.patch
          11 kB
          Mayank Bansal
        4. YARN-2933-4.patch
          14 kB
          Mayank Bansal
        5. YARN-2933-5.patch
          14 kB
          Mayank Bansal
        6. YARN-2933-6.patch
          12 kB
          Mayank Bansal
        7. YARN-2933-7.patch
          14 kB
          Mayank Bansal
        8. YARN-2933-8.patch
          14 kB
          Mayank Bansal
        9. YARN-2933-9.patch
          14 kB
          Mayank Bansal

        Activity

          People

            mayank_bansal Mayank Bansal
            leftnoteasy Wangda Tan
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: