[MAPREDUCE-3902] MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: applicationmaster, mrv2
• Assignee: Kannan Rajah
• Reporter: Arun Murthy
• Created: 23/Feb/12 15:09
• Updated: 08/Aug/16 17:04

Description

The MR AM is now in a great position to reuse containers across (map) tasks. This is something similar to JVM re-use we had in 0.20.x, but in a significantly better manner:

1. Consider data-locality when re-using containers
2. Consider the new shuffle - ensure that reduces fetch output of the whole container at once (i.e. all maps) : MAPREDUCE-4525

Issue Links

• is blocked by: YARN-75 RMContainer should handle a RELEASE event while RUNNING
• is related to: MAPREDUCE-4525 Combiner per node
• relates to: MAPREDUCE-4596 Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)

Sub-Tasks

• [MAPREDUCE-3902] TaskHeartbeatHandler should extends HeartbeatHandlerBase: Resolved
• [MAPREDUCE-3902] ScheduledRequests#remove should remove the elements from mapsHostMapping and mapsRackMapping: Open
• [MAPREDUCE-3902] Ensure not to launch container on blacklisted hosts: Resolved
• [MAPREDUCE-3902] Re-create ask list correctly in case of a temporary error in the AM-RM allocate call: Resolved
• [MAPREDUCE-3902] RMContainerAllocator#scheduleInterval should be configurable: Resolved
• [MAPREDUCE-3902] RMContainerAllocator#assign should be split into functions: Open
• [MAPREDUCE-3902] Re-wire AM Recovery: Resolved
• [MAPREDUCE-3902] Re-wire LocalContainerAllocator / UberAM: Resolved
• [MAPREDUCE-3902] Change AMContainerMap to extend AbstractService: Resolved
• [MAPREDUCE-3902] RMContainerAllocator should factor in nodes being blacklisted: Resolved
• [MAPREDUCE-3902] Disable AM blacklisting if #blacklistedNodes crosses the configured threshold: Resolved
• [MAPREDUCE-3902] Unit tests for AMContainer: Open
• [MAPREDUCE-3902] Unit tests for AMNode: Open
• [MAPREDUCE-3902] Reduce scheduling fixes, factor in MR-4437: Resolved
• [MAPREDUCE-3902] Statistics logging in the AM scheduler: Resolved
• [MAPREDUCE-3902] Fix and re-enable RMContaienrAllocator unit tests: Resolved
• [MAPREDUCE-3902] Handle the JobFinishedEvent correctly: Open
• [MAPREDUCE-3902] Container Launch should be independent of o.a.h.m.Task: Resolved
• [MAPREDUCE-3902] ContainerHeartbeatHandler should be pinged on a getTask call: Resolved
• [MAPREDUCE-3902] Use the configured shuffle port and application ACLs: Resolved
• [MAPREDUCE-3902] Handle a successful NM stop request: Resolved
• [MAPREDUCE-3902] re-enable disabled unit tests in mr-client-app2 module: Open