[YARN-6151] FS preemption does not consider child queues over fairshare if the parent is under

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0
• Component/s: fairscheduler
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Yufei Gu
• Reporter: Yufei Gu
• Created: 06/Feb/17 19:42
• Updated: 04/Apr/17 16:44
• Resolved: 10/Feb/17 04:24

Description

This is preemption bug happens before 2.8.0, which also described in YARN-3405.

Queue hierarchy described as below:

                  root
               /         \
       queue-1          queue-2       
          /      \
queue-1-1     queue-1-2

Assume cluster resource is 100 and all queues have same weights.

1. queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
2. When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but this doesn't happen because preemption happens top-down, queue-2 could be the preemption candidate as long as queue-2 is less needy than queue-1, and queue-2 doesn't exceed the fair share which means preemption won't happen.

We need to filter out queue-2 since it isn't a valid candidate.

Issue Links

• relates to: YARN-3405 FairScheduler's preemption cannot happen between sibling in some case
• links to: GitHub Pull Request #188