[MAPREDUCE-4833] Task can get stuck in FAIL_CONTAINER_CLEANUP

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.5
• Fix Version/s: 2.0.3-alpha, 0.23.6
• Component/s: applicationmaster, mrv2
• Target Version/s: 0.23.6
• Hadoop Flags: Reviewed
• Assignee: Robert Parker
• Reporter: Robert Joseph Evans
• Created: 29/Nov/12 20:53
• Updated: 15/Feb/13 13:10
• Resolved: 21/Dec/12 22:38

Description

If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck.