[MAPREDUCE-6749] MR AM should reuse containers for Map/Reduce Tasks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: applicationmaster, mrv2
• Assignee: Devaraj Kavali
• Reporter: Devaraj Kavali
• Created: 08/Aug/16 18:00
• Updated: 23/Feb/21 04:54

Description

It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.

Sub-Tasks

• Add MR Job Configurations for Containers reuse: Resolved
• Create ContainerRequestor interface and refactor RMContainerRequestor to use it: Resolved
• Implement RM Container Reuse Requestor to handle the reuse containers for resource requests: Resolved
• TaskAttemptImpl state changes for containers reuse: Patch Available
• ContainerLauncherImpl support for reusing the containers: Resolved
• YarnChild should wait for another task when reuse is enabled: Patch Available
• JobImpl state changes for containers reuse: Open
• Display only corresponding Task logs for each task attempt in AM and JHS Web UI: Open
• Containers never get reused as containersToReuse map gets cleared on makeRemoteRequest: Patch Available
• Job will hang if NM is restarted while its running: Patch Available