Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4305

Implement delay scheduling in capacity scheduler for improving data locality

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Capacity Scheduler data local tasks are about 40%-50% which is not good.
      While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster.

      I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality.
      http://radlab.cs.berkeley.edu/publication/308

      After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster.

      Thanks,
      Mayank

        Attachments

        1. MAPREDUCE-4305
          22 kB
          Mayank Bansal
        2. MAPREDUCE-4305-1.patch
          22 kB
          Mayank Bansal
        3. PATCH-MAPREDUCE-4305-MR1.patch
          43 kB
          Mayank Bansal
        4. PATCH-MAPREDUCE-4305-MR1-1.patch
          43 kB
          Mayank Bansal
        5. PATCH-MAPREDUCE-4305-MR1-2.patch
          39 kB
          Mayank Bansal
        6. PATCH-MAPREDUCE-4305-MR1-3.patch
          40 kB
          Mayank Bansal
        7. PATCH-MAPREDUCE-4305-MR1-6.patch
          35 kB
          Mayank Bansal
        8. PATCH-MAPREDUCE-4305-MR1-7.patch
          35 kB
          Mayank Bansal

          Issue Links

            Activity

              People

              • Assignee:
                mayank_bansal Mayank Bansal
                Reporter:
                mayank_bansal Mayank Bansal
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated: