[YARN-4108] CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: capacity scheduler
• Hadoop Flags: Reviewed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 03/Sep/15 00:01
• Updated: 14/Oct/19 15:38
• Resolved: 17/Mar/16 00:05

Description

This is sibling JIRA for YARN-2154. We should make sure container preemption is more effective.

Requirements::
1) Can handle case of user-limit preemption
2) Can handle case of resource placement requirements, such as: hard-locality (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1 and host[1-3])
3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).

Issue Links

• is duplicated by: YARN-4390 Do surgical preemption based on reserved container in CapacityScheduler
• relates to: YARN-2069 CS queue level preemption should respect user-limits
• relates to: YARN-2009 CapacityScheduler: Add intra-queue preemption for app priority support
• relates to: YARN-2113 Add cross-user preemption within CapacityScheduler's leaf-queue
• relates to: YARN-3510 Create an extension of ProportionalCapacityPreemptionPolicy which preempts a number of containers from each application in a way which respects fairness
• relates to: YARN-2154 FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request