[YARN-4690] Skip object allocation in FSAppAttempt#getResourceUsage when possible

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 12/Feb/16 07:07
• Updated: 06/Jan/17 08:04
• Resolved: 18/Feb/16 05:08

Description

YARN-2768 addresses an important bottleneck. Here is another similar instance where object allocation in Resources#subtract will slow down the fair scheduler's event processing thread.

org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java)
    org.apache.hadoop.yarn.util.Records.newRecord(Records.java)
    org.apache.hadoop.yarn.util.resource.Resources.createResource(Resources.java)
    org.apache.hadoop.yarn.util.resource.Resources.clone(Resources.java)
    org.apache.hadoop.yarn.util.resource.Resources.subtract(Resources.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.getResourceUsage(FSAppAttempt.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.getResourceUsage(FSLeafQueue.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
    java.util.TimSort.binarySort(TimSort.java)
    java.util.TimSort.sort(TimSort.java)
    java.util.TimSort.sort(TimSort.java)
    java.util.Arrays.sort(Arrays.java)
    java.util.Collections.sort(Collections.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
    org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
    org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)

One way to fix it is to return getCurrentConsumption() if there is no preemption which is the normal case. This means getResourceUsage method will return reference to FSAppAttempt's internal resource object. But that should be ok as getResourceUsage doesn't expect the caller to modify the object.