Hadoop YARN
  1. Hadoop YARN
  2. YARN-80

Support delay scheduling for node locality in MR2's capacity scheduler

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha, 0.23.6
    • Component/s: capacityscheduler
    • Labels:
      None

      Description

      The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.

      1. YARN-80.patch
        5 kB
        Arun C Murthy
      2. YARN-80.patch
        8 kB
        Arun C Murthy

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Assignee Todd Lipcon [ tlipcon ]
          Arun C Murthy made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Key MAPREDUCE-3210 YARN-80
          Affects Version/s 0.23.0 [ 12315570 ]
          Target Version/s 0.23.0 [ 12315570 ]
          Component/s capacityscheduler [ 12319324 ]
          Component/s mrv2 [ 12314301 ]
          Arun C Murthy made changes -
          Assignee Arun C Murthy [ acmurthy ]
          Arun C Murthy made changes -
          Attachment YARN-80.patch [ 12543531 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment YARN-80.patch [ 12543823 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322548 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Harsh J made changes -
          Link This issue is related too MAPREDUCE-4305 [ MAPREDUCE-4305 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Harsh J made changes -
          Link This issue supercedes YARN-284 [ YARN-284 ]
          Thomas Graves made changes -
          Fix Version/s 0.23.6 [ 12323501 ]
          Gavin made changes -
          Link This issue is related to MAPREDUCE-4305 [ MAPREDUCE-4305 ]
          Gavin made changes -
          Link This issue is related to MAPREDUCE-4305 [ MAPREDUCE-4305 ]

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development