Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2768

Avoid cloning Resource in FSAppAttempt#updateDemand

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See the attached picture of profiling result. The clone of Resource object within Resources.multiply() takes up *85%* (19.2 / 22.6) CPU time of the function FairScheduler.update().

      The code of FSAppAttempt.updateDemand:

      public void updateDemand() {
          demand = Resources.createResource(0);
          // Demand is current consumption plus outstanding requests
          Resources.addTo(demand, app.getCurrentConsumption());
      
          // Add up outstanding resource requests
          synchronized (app) {
            for (Priority p : app.getPriorities()) {
              for (ResourceRequest r : app.getResourceRequests(p).values()) {
                Resource total = Resources.multiply(r.getCapability(), r.getNumContainers());
                Resources.addTo(demand, total);
              }
            }
          }
        }
      

      The code of Resources.multiply:

      public static Resource multiply(Resource lhs, double by) {
          return multiplyTo(clone(lhs), by);
      }
      

      The clone could be skipped by directly update the value of this.demand.

        Attachments

        1. profiling_FairScheduler_update.png
          69 kB
          Hong Zhiguo
        2. YARN-2768.patch
          2 kB
          Hong Zhiguo

          Activity

            People

            • Assignee:
              zhiguohong Hong Zhiguo
              Reporter:
              zhiguohong Hong Zhiguo
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: