[YARN-7469] Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
• Fix Version/s: 2.8.3, 3.0.0, 3.1.0, 2.10.0, 2.9.1
• Component/s: capacity scheduler, yarn
• Hadoop Flags: Reviewed
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 09/Nov/17 22:25
• Updated: 14/Oct/19 15:37
• Resolved: 16/Nov/17 17:32

Description

Queue Configuration:

• Total Memory: 20GB
• 2 Queues
  • Queue1
    • Memory: 10GB
    • MULP: 10%
    • ULF: 2.0
• Minimum Container Size: 0.5GB

Use Case:

• User1 submits app1 to Queue1 and consumes 20GB
• User2 submits app2 to Queue1 and requests 7.5GB
• Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those resources to User2
• User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 container for the AM.
• Preemption monitor never preempts a container.

Issue Links

• relates to: YARN-7496 CS Intra-queue preemption user-limit calculations are not in line with LeafQueue user-limit calculations
• relates to: YARN-7501 Capacity Scheduler Intra-queue preemption should have a "dead zone" around user limit