Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3902

MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: applicationmaster, mrv2
    • Labels:
      None

      Description

      The MR AM is now in a great position to reuse containers across (map) tasks. This is something similar to JVM re-use we had in 0.20.x, but in a significantly better manner:

      1. Consider data-locality when re-using containers
      2. Consider the new shuffle - ensure that reduces fetch output of the whole container at once (i.e. all maps) : MAPREDUCE-4525
      1. MAPREDUCE-3902.patch
        74 kB
        Arun C Murthy
      2. MAPREDUCE-3902.2.patch
        55 kB
        Tsuyoshi Ozawa
      3. AM_ContainerRefactor.pdf
        151 kB
        Siddharth Seth
      4. AMContainerRefactorNotes.pdf
        58 kB
        Siddharth Seth

        Issue Links

        1.
        [MAPREDUCE-3902] TaskHeartbeatHandler should extends HeartbeatHandlerBase Sub-task Resolved Tsuyoshi Ozawa
         
        2. [MAPREDUCE-3902] ScheduledRequests#remove should remove the elements from mapsHostMapping and mapsRackMapping Sub-task Open Tsuyoshi Ozawa
         
        3.
        [MAPREDUCE-3902] Ensure not to launch container on blacklisted hosts Sub-task Resolved Tsuyoshi Ozawa
         
        4.
        [MAPREDUCE-3902] Re-create ask list correctly in case of a temporary error in the AM-RM allocate call Sub-task Resolved Siddharth Seth
         
        5.
        [MAPREDUCE-3902] RMContainerAllocator#scheduleInterval should be configurable Sub-task Resolved Tsuyoshi Ozawa
         
        6. [MAPREDUCE-3902] RMContainerAllocator#assign should be split into functions Sub-task Open Tsuyoshi Ozawa
         
        7.
        [MAPREDUCE-3902] Re-wire AM Recovery Sub-task Resolved Siddharth Seth
         
        8.
        [MAPREDUCE-3902] Re-wire LocalContainerAllocator / UberAM Sub-task Resolved Siddharth Seth
         
        9.
        [MAPREDUCE-3902] Change AMContainerMap to extend AbstractService Sub-task Resolved Tsuyoshi Ozawa
         
        10.
        [MAPREDUCE-3902] RMContainerAllocator should factor in nodes being blacklisted Sub-task Resolved Siddharth Seth
         
        11.
        [MAPREDUCE-3902] Disable AM blacklisting if #blacklistedNodes crosses the configured threshold Sub-task Resolved Siddharth Seth
         
        12. [MAPREDUCE-3902] Unit tests for AMContainer Sub-task Open Unassigned
         
        13. [MAPREDUCE-3902] Unit tests for AMNode Sub-task Open Unassigned
         
        14.
        [MAPREDUCE-3902] Reduce scheduling fixes, factor in MR-4437 Sub-task Resolved Siddharth Seth
         
        15.
        [MAPREDUCE-3902] Statistics logging in the AM scheduler Sub-task Resolved Siddharth Seth
         
        16.
        [MAPREDUCE-3902] Fix and re-enable RMContaienrAllocator unit tests Sub-task Resolved Siddharth Seth
         
        17. [MAPREDUCE-3902] Handle the JobFinishedEvent correctly Sub-task Open Siddharth Seth
         
        18.
        [MAPREDUCE-3902] Container Launch should be independent of o.a.h.m.Task Sub-task Resolved Siddharth Seth
         
        19.
        [MAPREDUCE-3902] ContainerHeartbeatHandler should be pinged on a getTask call Sub-task Resolved Siddharth Seth
         
        20.
        [MAPREDUCE-3902] Use the configured shuffle port and application ACLs Sub-task Resolved Siddharth Seth
         
        21.
        [MAPREDUCE-3902] Handle a successful NM stop request Sub-task Resolved Siddharth Seth
         
        22. [MAPREDUCE-3902] re-enable disabled unit tests in mr-client-app2 module Sub-task Open Siddharth Seth
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Kannan Rajah
              Reporter:
              Arun C Murthy
            • Votes:
              1 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

              • Created:
                Updated:

                Development