Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5139

[Umbrella] Move YARN scheduler towards global scheduler

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Existing YARN scheduler is based on node heartbeat. This can lead to sub-optimal decisions because scheduler can only look at one node at the time when scheduling resources.

      Pseudo code of existing scheduling logic looks like:

      for node in allNodes:
         Go to parentQueue
            Go to leafQueue
              for application in leafQueue.applications:
                 for resource-request in application.resource-requests
                    try to schedule on node
      

      Considering future complex resource placement requirements, such as node constraints (give me "a && b || c") or anti-affinity (do not allocate HBase regionsevers and Storm workers on the same host), we may need to consider moving YARN scheduler towards global scheduling.

        Attachments

        1. wip-1.YARN-5139.patch
          76 kB
          Wangda Tan
        2. wip-2.YARN-5139.patch
          95 kB
          Wangda Tan
        3. wip-3.YARN-5139.patch
          133 kB
          Wangda Tan
        4. YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf
          245 kB
          Wangda Tan
        5. YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf
          285 kB
          Wangda Tan
        6. wip-4.YARN-5139.patch
          447 kB
          Wangda Tan
        7. Explanantions of Global Scheduling (YARN-5139) Implementation.pdf
          283 kB
          Wangda Tan
        8. YARN-5139-Concurrent-scheduling-performance-report.pdf
          96 kB
          Wangda Tan
        9. wip-5.YARN-5139.patch
          386 kB
          Wangda Tan
        10. YARN-5139.000.patch
          402 kB
          Wangda Tan

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                5 Vote for this issue
                Watchers:
                83 Start watching this issue

                Dates

                • Created:
                  Updated: