[YARN-4752] FairScheduler should preempt for a ResourceRequest and all preempted containers should be on the same node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha2
• Component/s: fairscheduler
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 02/Mar/16 01:28
• Updated: 14/Aug/18 02:33
• Resolved: 09/Dec/16 23:30

Description

A number of issues have been reported with respect to preemption in FairScheduler along the lines of:

1. FairScheduler preempts resources from nodes even if the resultant free resources cannot fit the incoming request.
2. Preemption doesn't preempt from sibling queues
3. Preemption doesn't preempt from sibling apps under the same queue that is over its fairshare
4. ...

Filing this umbrella JIRA to group all the issues together and think of a comprehensive solution.

Issue Links

• breaks: YARN-6144 FairScheduler: preempted resources can become negative
• incorporates: YARN-4931 Preempted resources go back to the same application
• incorporates: YARN-2154 FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
• incorporates: YARN-3405 FairScheduler's preemption cannot happen between sibling in some case
• incorporates: YARN-3997 An Application requesting multiple core containers can't preempt running application made of single core containers
• incorporates: YARN-2457 FairScheduler: Handle preemption to help starved parent queues
• incorporates: YARN-3054 Preempt policy in FairScheduler may cause mapreduce job never finish
• incorporates: YARN-3121 FairScheduler preemption metrics
• incorporates: YARN-3902 Fair scheduler preempts ApplicationMaster
• incorporates: YARN-4120 FSAppAttempt.getResourceUsage() should not take preemptedResource into account
• incorporates: YARN-4133 Containers to be preempted leak in FairScheduler preemption logic.
• incorporates: YARN-4333 Fair scheduler should support preemption within a leaf queue
• is blocked by: YARN-5181 ClusterNodeTracker: add method to get list of nodes matching a specific resourceName
• is duplicated by: YARN-6076 Backport YARN-4752 (FS preemption changes) to branch-2
• is related to: YARN-8655 FairScheduler: FSStarvedApps is not thread safe

Sub-Tasks

• Preempt containers (all on one node) to meet the requirement of starved applications: Resolved
• Drop left-over preemption-related code and clean up method visibilities in the Schedulable hierarchy: Resolved
• Verify identification of starved applications: Resolved
• Verify fairshare and minshare preemption: Resolved
• Cleanup YARN-4752 for merge: Resolved