[YARN-8668] Inconsistency between capacity and fair scheduler in the aspect of computing node available resource

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Labels: capacityscheduler
• Assignee: Yeliang Cang
• Reporter: Yeliang Cang
• Created: 15/Aug/18 09:33
• Updated: 16/Aug/18 02:54
• Resolved: 16/Aug/18 02:54

Description

We have observed that given capacityScheduler and defaultResourceCalculor,  when there are many memory resources in a node, running heavy workload, then the available vcores of this node will be negative!

I noticed that in capacityScheduler.java, use code below to calculate the available resources for allocating containers:

if (calculator.computeAvailableContainers(Resources
 .add(node.getUnallocatedResource(), node.getTotalKillableResources()),
 minimumAllocation) <= 0) {
 if (LOG.isDebugEnabled()) {
 LOG.debug("This node or this node partition doesn't have available or"
 + "killable resource");
 }

while in fairscheduler FsAppAttempt.java, similar code was found:

// Can we allocate a container on this node?
if (Resources.fitsIn(capability, available)) {

...

}

Why is the inconsistency? I think we should use Resources.fitsIn(smaller,bigger) instead in capacityScheduler !!!