[YARN-4822] Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 15/Mar/16 18:50
• Updated: 14/Oct/19 15:37
• Resolved: 30/Mar/16 19:47

Description

Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to select candidates to be preempted (based on FIFO order of applications/containers). It's not a simple to add new candidate-selection logics, such as preemption for large container, intra-queeu fairness/policy, etc.

In this JIRA, I propose to do following changes:
1) Cleanup code bases, consolidate current logic into 3 stages:

• Compute ideal sharing of queues
• Select to-be-preempt candidates
• Send preemption/kill events to scheduler

2) Add a new interface: PreemptionCandidatesSelectionPolicy for above "select to-be-preempt candidates" part. Move existing how to select candidates logics to FifoPreemptionCandidatesSelectionPolicy.
3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to select candidates, and later PreemptionCandidatesSelectionPolicy can make decisions according to already selected candidates and pre-computed queue ideal shares of resources.

Issue Links

• is depended upon by: YARN-4390 Do surgical preemption based on reserved container in CapacityScheduler