Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6242 [Umbrella] Miscellaneous Scheduler Performance Improvements
  3. YARN-6775

CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.1, 3.0.0-alpha3
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: capacityscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are several things in assignContainers() that are done multiple times even though the result cannot change (canAssignToUser, canAssignToQueue). Add some local caching to take advantage of this fact.

      Will post patch shortly. Patch includes a simple throughput test that demonstrates when we have users at their user-limit, the number of NodeUpdateSchedulerEvents we can process can be improved from 13K/sec to 50K/sec.

        Attachments

        1. rmeventprocbusy.png
          39 kB
          Nathan Roberts
        2. rpcprocessingtimeschedulerport.png
          38 kB
          Nathan Roberts
        3. YARN-6775.001.patch
          22 kB
          Nathan Roberts
        4. YARN-6775.002.patch
          22 kB
          Nathan Roberts
        5. YARN-6775.branch-2.002.patch
          21 kB
          Nathan Roberts
        6. YARN-6775.branch-2.003.patch
          21 kB
          Wangda Tan
        7. YARN-6775.branch-2.8.002.patch
          22 kB
          Nathan Roberts

          Issue Links

            Activity

              People

              • Assignee:
                nroberts Nathan Roberts
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: