Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4305

Implement delay scheduling in capacity scheduler for improving data locality

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Capacity Scheduler data local tasks are about 40%-50% which is not good.
      While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster.

      I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality.
      http://radlab.cs.berkeley.edu/publication/308

      After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster.

      Thanks,
      Mayank

      1. PATCH-MAPREDUCE-4305-MR1-7.patch
        35 kB
        Mayank Bansal
      2. PATCH-MAPREDUCE-4305-MR1-6.patch
        35 kB
        Mayank Bansal
      3. PATCH-MAPREDUCE-4305-MR1-3.patch
        40 kB
        Mayank Bansal
      4. PATCH-MAPREDUCE-4305-MR1-2.patch
        39 kB
        Mayank Bansal
      5. PATCH-MAPREDUCE-4305-MR1-1.patch
        43 kB
        Mayank Bansal
      6. PATCH-MAPREDUCE-4305-MR1.patch
        43 kB
        Mayank Bansal
      7. MAPREDUCE-4305-1.patch
        22 kB
        Mayank Bansal
      8. MAPREDUCE-4305
        22 kB
        Mayank Bansal

        Issue Links

          Activity

          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1-7.patch [ 12582002 ]
          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1-6.patch [ 12581964 ]
          Gavin made changes -
          Link This issue relates to YARN-80 [ YARN-80 ]
          Gavin made changes -
          Link This issue relates to YARN-80 [ YARN-80 ]
          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1-3.patch [ 12569603 ]
          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1-2.patch [ 12569290 ]
          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1-1.patch [ 12564066 ]
          Mayank Bansal made changes -
          Attachment PATCH-MAPREDUCE-4305-MR1.patch [ 12563866 ]
          Harsh J made changes -
          Link This issue relates YARN-80 [ YARN-80 ]
          Mayank Bansal made changes -
          Attachment MAPREDUCE-4305-1.patch [ 12531025 ]
          Mayank Bansal made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-4305 [ 12530881 ]
          Mayank Bansal created issue -

            People

            • Assignee:
              Mayank Bansal
              Reporter:
              Mayank Bansal
            • Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:

                Development