Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4690

Skip object allocation in FSAppAttempt#getResourceUsage when possible

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      YARN-2768 addresses an important bottleneck. Here is another similar instance where object allocation in Resources#subtract will slow down the fair scheduler's event processing thread.

      org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java)
          org.apache.hadoop.yarn.util.Records.newRecord(Records.java)
          org.apache.hadoop.yarn.util.resource.Resources.createResource(Resources.java)
          org.apache.hadoop.yarn.util.resource.Resources.clone(Resources.java)
          org.apache.hadoop.yarn.util.resource.Resources.subtract(Resources.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt.getResourceUsage(FSAppAttempt.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.getResourceUsage(FSLeafQueue.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FairSharePolicy$FairShareComparator.compare(FairSharePolicy.java)
          java.util.TimSort.binarySort(TimSort.java)
          java.util.TimSort.sort(TimSort.java)
          java.util.TimSort.sort(TimSort.java)
          java.util.Arrays.sort(Arrays.java)
          java.util.Collections.sort(Collections.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java)
          org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
          org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.handle(ResourceSchedulerWrapper.java)
      

      One way to fix it is to return getCurrentConsumption() if there is no preemption which is the normal case. This means getResourceUsage method will return reference to FSAppAttempt's internal resource object. But that should be ok as getResourceUsage doesn't expect the caller to modify the object.

        Attachments

          Activity

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: