Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There're 3 stages in ProportionalCapacityPreemptionPolicy,

      1. Recursively calculate ideal_assigned for queue. This is depends on available resource, resource used/pending in each queue and guaranteed capacity of each queue.
      2. Mark to-be preempted containers: For each over-satisfied queue, it will mark some containers will be preempted.
      3. Notify scheduler about to-be preempted container.

      We need respect labels in the cluster for both #1 and #2:
      For #1, when calculating ideal_assigned for each queue, we need get by-partition-ideal-assigned according to queue's guaranteed/maximum/used/pending resource on specific partition.
      For #2, when we make decision about whether we need preempt a container, we need make sure, resource this container is possibly usable by a queue which is under-satisfied and has pending resource.

      In addition, we need to handle ignore_partition_exclusivity case, when we need to preempt containers from a queue's partition, we will first preempt ignore_partition_exclusivity allocated containers.

        Attachments

        1. STALED-YARN-2498.zip
          63 kB
          Wangda Tan
        2. YARN-2498.1.patch
          114 kB
          Wangda Tan
        3. YARN-2498.2.patch
          117 kB
          Wangda Tan
        4. YARN-2498.3.patch
          117 kB
          Wangda Tan
        5. YARN-2498.4.patch
          119 kB
          Wangda Tan

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: