[YARN-1961] Fair scheduler preemption doesn't work for non-leaf queues

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: fairscheduler, scheduler
• Labels: scheduler
• Assignee: Ashwin Shankar
• Reporter: Ashwin Shankar
• Created: 18/Apr/14 19:17
• Updated: 11/Mar/16 21:53
• Resolved: 11/Mar/16 21:53

Description

Setting minResources and minSharePreemptionTimeout to a non-leaf queue doesn't cause preemption to happen when that non-leaf queue is below minResources and there are outstanding demands in that non-leaf queue.
Here is an example fs allocation config(partial) :

<queue name="abc">
  <minResources>3072 mb,0 vcores</minResources>
  <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
    <queue name="childabc1">
    </queue>
    <queue name="childabc2">
    </queue>
 </queue>
 

With the above configs,preemption doesn't seem to happen if queue abc is below minShare and it has outstanding unsatisfied demands from apps in its child queues. Ideally in such cases we would like preemption to kick off and reclaim resources from other queues(not under queue abc).

Looking at the code it seems like preemption checks for starvation only at the leaf queue level and not at the parent level.

FairScheduler.java

boolean isStarvedForMinShare(FSLeafQueue sched)
boolean isStarvedForFairShare(FSLeafQueue sched)

This affects our use case where we have a parent queue with probably a 100 unconfigured leaf queues under it.We want to give a minshare to the parent queue to protect all the leaf queues under it,but we cannot do it due to this bug.