Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6242 [Umbrella] Miscellaneous Scheduler Performance Improvements
  3. YARN-6361

FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy. Most of the time is spent in FairShareComparator.compare. We could improve this by doing the calculations outside the sort loop (O(n)) and we sorted by a fixed number inside instead O(n*log(n)). This could be an performance issue when there are huge number of applications in a single queue. The attachments shows the performance impact when there are 10k applications in one queue.

      Attachments

        1. YARN-6361.001.patch
          19 kB
          YunFan Zhou
        2. threads.png
          34 kB
          Miklos Szegedi
        3. dispatcherthread.png
          68 kB
          Miklos Szegedi

        Activity

          People

            daemon YunFan Zhou
            miklos.szegedi@cloudera.com Miklos Szegedi
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated: