[YARN-2498] Respect labels in preemption policy of capacity scheduler for inter-queue preemption

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: resourcemanager
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 03/Sep/14 08:56
• Updated: 14/Oct/19 15:37
• Resolved: 25/Apr/15 00:05

Description

There're 3 stages in ProportionalCapacityPreemptionPolicy,

1. Recursively calculate ideal_assigned for queue. This is depends on available resource, resource used/pending in each queue and guaranteed capacity of each queue.
2. Mark to-be preempted containers: For each over-satisfied queue, it will mark some containers will be preempted.
3. Notify scheduler about to-be preempted container.

We need respect labels in the cluster for both #1 and #2:
For #1, when calculating ideal_assigned for each queue, we need get by-partition-ideal-assigned according to queue's guaranteed/maximum/used/pending resource on specific partition.
For #2, when we make decision about whether we need preempt a container, we need make sure, resource this container is possibly usable by a queue which is under-satisfied and has pending resource.

In addition, we need to handle ignore_partition_exclusivity case, when we need to preempt containers from a queue's partition, we will first preempt ignore_partition_exclusivity allocated containers.

Issue Links

• blocks: YARN-3515 Support intra-queue non-exclusive container allocation within a LeafQueue
• is depended upon by: YARN-3515 Support intra-queue non-exclusive container allocation within a LeafQueue