Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5139

[Umbrella] Move YARN scheduler towards global scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

      Pseudo code of existing scheduling logic looks like:

      for node in allNodes:
         Go to parentQueue
            Go to leafQueue
              for application in leafQueue.applications:
                 for resource-request in application.resource-requests
                    try to schedule on node
      

      Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

      Attachments

        1. Explanantions of Global Scheduling (YARN-5139) Implementation.pdf
          283 kB
          Wangda Tan
        2. wip-1.YARN-5139.patch
          76 kB
          Wangda Tan
        3. wip-2.YARN-5139.patch
          95 kB
          Wangda Tan
        4. wip-3.YARN-5139.patch
          133 kB
          Wangda Tan
        5. wip-4.YARN-5139.patch
          447 kB
          Wangda Tan
        6. wip-5.YARN-5139.patch
          386 kB
          Wangda Tan
        7. YARN-5139.000.patch
          402 kB
          Wangda Tan
        8. YARN-5139-Concurrent-scheduling-performance-report.pdf
          96 kB
          Wangda Tan
        9. YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf
          245 kB
          Wangda Tan
        10. YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf
          285 kB
          Wangda Tan

        Issue Links

          Activity

            People

              leftnoteasy Wangda Tan
              leftnoteasy Wangda Tan
              Votes:
              4 Vote for this issue
              Watchers:
              86 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h