Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-80

Support delay scheduling for node locality in MR2's capacity scheduler

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.2-alpha, 0.23.6
    • capacityscheduler
    • None

    Description

      The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.

      Attachments

        1. YARN-80.patch
          8 kB
          Arun Murthy
        2. YARN-80.patch
          5 kB
          Arun Murthy

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            acmurthy Arun Murthy
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            26 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment