[YARN-6193] FairScheduler might not trigger preemption when using DRF

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: fairscheduler
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 14/Feb/17 21:54
• Updated: 21/Apr/17 21:51
• Resolved: 17/Feb/17 22:44

Description

FSAppAttempt#canContainerBePreempted verifies preempting a container doesn't lead to new starvation (Resources.fitsIn). When using DRF, this leads to verifying both resources instead of just the dominant resource. Ideally, the check should be based on policy.

Note that current implementation of DominantResourceFairnessPolicy#checkIfUsageOverFairShare is broken.

Issue Links

• is blocked by: YARN-6163 FS Preemption is a trickle for severely starved applications