[YARN-4120] FSAppAttempt.getResourceUsage() should not take preemptedResource into account

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: fairscheduler
• Assignee: Unassigned
• Reporter: Xianyin Xin
• Created: 06/Sep/15 08:48
• Updated: 31/Aug/16 03:49
• Resolved: 31/Aug/16 03:45

Description

When compute resource usage for Schedulables, the following code is envolved,
FSAppAttempt.getResourceUsage,

public Resource getResourceUsage() {
  return Resources.subtract(getCurrentConsumption(), getPreemptedResources());
}

and this value is aggregated to FSLeafQueues and FSParentQueues. In my opinion, taking preemptedResource into account here is not reasonable, there are two main reasons,

1. it is something in future, i.e., even though these resources are marked as preempted, it is currently used by app, and these resources will be subtracted from currentCosumption once the preemption is finished. it's not reasonable to make arrange for it ahead of time.
2. there's another problem here, consider following case,

               root
              /    \
         queue1   queue2
         /    \
   queue1.3, queue1.4
   

   suppose queue1.3 need resource and it can preempt resources from queue1.4, the preemption happens in the interior of queue1. But when compute resource usage of queue1, queue1.resourceUsage = it's_current_resource_usage - preemption according to the current code, which is unfair to queue2 when doing resource allocating.

Issue Links

• is part of: YARN-4752 FairScheduler should preempt for a ResourceRequest and all preempted containers should be on the same node
• is related to: YARN-4090 Make Collections.sort() more efficient by caching resource usage
• relates to: YARN-4134 FairScheduler preemption stops at queue level that all child queues are not over their fairshare