Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-80

Support delay scheduling for node locality in MR2's capacity scheduler

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha, 0.23.6
    • Component/s: capacityscheduler
    • Labels:
      None

      Description

      The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.

        Attachments

        1. YARN-80.patch
          8 kB
          Arun C Murthy
        2. YARN-80.patch
          5 kB
          Arun C Murthy

          Issue Links

            Activity

              People

              • Assignee:
                acmurthy Arun C Murthy
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: