[YARN-7149] Cross-queue preemption sometimes starves an underserved queue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.0-alpha3
• Fix Version/s: 2.9.0, 3.0.0-beta1, 3.1.0
• Component/s: capacity scheduler
• Hadoop Flags: Reviewed
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 01/Sep/17 19:52
• Updated: 14/Oct/19 15:38
• Resolved: 16/Sep/17 04:33

Description

In branch 2 and trunk, I am consistently seeing some use cases where cross-queue preemption does not happen when it should. I do not see this in branch-2.8.

Use Case:

	Size	Minimum Container Size
MyCluster	20 GB	0.5 GB

Queue Name	Capacity	Absolute Capacity	Minimum User Limit Percent (MULP)	User Limit Factor (ULF)
Q1	50% = 10 GB	100% = 20 GB	10% = 1 GB	2.0
Q2	50% = 10 GB	100% = 20 GB	10% = 1 GB	2.0

• User1 launches App1 in Q1 and consumes all resources (20 GB)
• User2 launches App2 in Q2 and requests 10 GB
• Note: containers are 0.5 GB.
• Preemption monitor kills 2 containers (equals 1 GB) from App1 in Q1.
• Capacity Scheduler assigns 2 containers (equals 1 GB) to App2 in Q2.
• No more containers are ever preempted, even though Q2 is far underserved