[YARN-3453] Fair Scheduler: Parts of preemption logic uses DefaultResourceCalculator even in DRF mode causing thrashing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fairscheduler
• Assignee: Arun Suresh
• Reporter: Ashwin Shankar
• Created: 06/Apr/15 20:43
• Updated: 01/Nov/17 22:00
• Resolved: 17/Jul/15 11:17

Description

There are two places in preemption code flow where DefaultResourceCalculator is used, even in DRF mode.
Which basically results in more resources getting preempted than needed, and those extra preempted containers aren’t even getting to the “starved” queue since scheduling logic is based on DRF's Calculator.

Following are the two places :
1.

FSLeafQueue.java

private boolean isStarved(Resource share)

A queue shouldn’t be marked as “starved” if the dominant resource usage
is >= fair/minshare.

2.

FairScheduler.java

protected Resource resToPreempt(FSLeafQueue sched, long curTime)

--------------------------------------------------------------

One more thing that I believe needs to change in DRF mode is : during a preemption round,if preempting a few containers results in satisfying needs of a resource type, then we should exit that preemption round, since the containers that we just preempted should bring the dominant resource usage to min/fair share.

Issue Links

• is related to: YARN-3405 FairScheduler's preemption cannot happen between sibling in some case