Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2768

Avoid cloning Resource in FSAppAttempt#updateDemand

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • fairscheduler
    • None
    • Reviewed

    Description

      See the attached picture of profiling result. The clone of Resource object within Resources.multiply() takes up *85%* (19.2 / 22.6) CPU time of the function FairScheduler.update().

      The code of FSAppAttempt.updateDemand:

      public void updateDemand() {
          demand = Resources.createResource(0);
          // Demand is current consumption plus outstanding requests
          Resources.addTo(demand, app.getCurrentConsumption());
      
          // Add up outstanding resource requests
          synchronized (app) {
            for (Priority p : app.getPriorities()) {
              for (ResourceRequest r : app.getResourceRequests(p).values()) {
                Resource total = Resources.multiply(r.getCapability(), r.getNumContainers());
                Resources.addTo(demand, total);
              }
            }
          }
        }
      

      The code of Resources.multiply:

      public static Resource multiply(Resource lhs, double by) {
          return multiplyTo(clone(lhs), by);
      }
      

      The clone could be skipped by directly update the value of this.demand.

      Attachments

        1. YARN-2768.patch
          2 kB
          Hong Zhiguo
        2. profiling_FairScheduler_update.png
          69 kB
          Hong Zhiguo

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zhiguohong Hong Zhiguo
            zhiguohong Hong Zhiguo
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment