Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11202

Optimize ClientRMService.getApplications

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • yarn
    • Reviewed

    Description

      YARN-11114 introduced an extra check in ClientRMService.getApplications() where it calculates a map (queue to a list of rmapps) inside a while loop which could have been calculated once before the while loop (which iterates over the apps).

      Code walk-through:

      queues:
       [q1, q2, q3]
      apps:                 (apps running/non-running)
       [a1, a2, a3, a4, a5, a6, a7, a8, a9]
      
      inside the while cycle:
        application:         (current iterator)
         [a5]
        foundApps:           (this is re-calculated every time, running apps)
          q1: [a1, a2]
          q2: [a4, a5, a6]
          q3: [a7, a9]
        runningAppsByQueues: (foundApps is already filtered by queues ... basically this concatenates the lists with extra steps)
          [a1, a2, a4, a5, a6, a7, a9]
        runningAppsById:
          [a5]
      

      I think this could be simplified significantly, basically we just need a set of running app-ids (filtered by queues), which is calculated before the while loop, once / request. Then inside the while loop:

      - if (!queues.contains(application.getQueue())) {
      + if (!runningApps.contains(application.getApplicationId()) && !queues.contains(application.getQueue())) {
      

      The tests pass with this fix.

      There is a potential bug inside the queryApplicationsByQueues, between the following time range: a new app is already in a queue, but not in the apps that was queried before:

      apps = rmContext.getRMApps();
      ...
      scheduler.getAppsInQueue(queue);

      Attachments

        Issue Links

          Activity

            People

              tdomok Tamas Domok
              tdomok Tamas Domok
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m