Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6242 [Umbrella] Miscellaneous Scheduler Performance Improvements
  3. YARN-6361

FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy. Most of the time is spent in FairShareComparator.compare. We could improve this by doing the calculations outside the sort loop (O(n)) and we sorted by a fixed number inside instead O(n*log(n)). This could be an performance issue when there are huge number of applications in a single queue. The attachments shows the performance impact when there are 10k applications in one queue.

        Attachments

        1. YARN-6361.001.patch
          19 kB
          YunFan Zhou
        2. threads.png
          34 kB
          Miklos Szegedi
        3. dispatcherthread.png
          68 kB
          Miklos Szegedi

          Activity

            People

            • Assignee:
              daemon YunFan Zhou
              Reporter:
              miklos.szegedi@cloudera.com Miklos Szegedi
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: