Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2492 (Clone of YARN-796) Allow for (admin) labels on nodes and resource-requests
  3. YARN-2498

Respect labels in preemption policy of capacity scheduler for inter-queue preemption

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • resourcemanager
    • None
    • Reviewed

    Description

      There're 3 stages in ProportionalCapacityPreemptionPolicy,

      1. Recursively calculate ideal_assigned for queue. This is depends on available resource, resource used/pending in each queue and guaranteed capacity of each queue.
      2. Mark to-be preempted containers: For each over-satisfied queue, it will mark some containers will be preempted.
      3. Notify scheduler about to-be preempted container.

      We need respect labels in the cluster for both #1 and #2:
      For #1, when calculating ideal_assigned for each queue, we need get by-partition-ideal-assigned according to queue's guaranteed/maximum/used/pending resource on specific partition.
      For #2, when we make decision about whether we need preempt a container, we need make sure, resource this container is possibly usable by a queue which is under-satisfied and has pending resource.

      In addition, we need to handle ignore_partition_exclusivity case, when we need to preempt containers from a queue's partition, we will first preempt ignore_partition_exclusivity allocated containers.

      Attachments

        1. YARN-2498.4.patch
          119 kB
          Wangda Tan
        2. YARN-2498.3.patch
          117 kB
          Wangda Tan
        3. YARN-2498.2.patch
          117 kB
          Wangda Tan
        4. YARN-2498.1.patch
          114 kB
          Wangda Tan
        5. STALED-YARN-2498.zip
          63 kB
          Wangda Tan

        Issue Links

          Activity

            People

              leftnoteasy Wangda Tan
              leftnoteasy Wangda Tan
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: