Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-961

ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Tags:
      fb

      Description

      Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network usage in that machine. The amount of resources currently used on each task tracker is being fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

      1. HIVE-961.patch
        127 kB
        Scott Chen
      2. MAPREDUCE-961-v2.patch
        139 kB
        Scott Chen
      3. MAPREDUCE-961-v3.patch
        51 kB
        Scott Chen
      4. MAPREDUCE-961-v4.patch
        27 kB
        Scott Chen
      5. ResourceScheduling.pdf
        338 kB
        Scott Chen

        Issue Links

          Activity

          dhruba borthakur created issue -
          dhruba borthakur made changes -
          Field Original Value New Value
          Tags fb
          dhruba borthakur made changes -
          Link This issue is blocked by MAPREDUCE-944 [ MAPREDUCE-944 ]
          Chris Douglas made changes -
          Tags fb
          Chris Douglas made changes -
          Tags fb
          Scott Chen made changes -
          Attachment HIVE-961.patch [ 12421891 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-961-v2.patch [ 12423355 ]
          Scott Chen made changes -
          Link This issue is related to MAPREDUCE-1167 [ MAPREDUCE-1167 ]
          Scott Chen made changes -
          Assignee dhruba borthakur [ dhruba ] Scott Chen [ schen ]
          Scott Chen made changes -
          Attachment MAPREDUCE-961-v3.patch [ 12424390 ]
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-961-v4.patch [ 12424880 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-961-v4.patch [ 12424880 ]
          Scott Chen made changes -
          Attachment MAPREDUCE-961-v4.patch [ 12424888 ]
          Scott Chen made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Scott Chen made changes -
          Attachment ResourceScheduling.pdf [ 12425873 ]
          Scott Chen made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314184 ]

            People

            • Assignee:
              Scott Chen
              Reporter:
              dhruba borthakur
            • Votes:
              3 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:

                Development