[MAPREDUCE-4751] AM stuck in KILL_WAIT for days

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.2-alpha
• Fix Version/s: 2.0.3-alpha, 0.23.5
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Ravi Prakash
• Created: 17/Oct/12 20:28
• Updated: 03/Sep/14 23:17
• Resolved: 12/Nov/12 16:57

Description

We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a few maps running. All these maps were scheduled on nodes which are now in the RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state

Issue Links

• duplicates: MAPREDUCE-4744 Application Master is running forever when the TaskAttempt gets TA_KILL event at the state SUCCESS_CONTAINER_CLEANUP
• duplicates: MAPREDUCE-4745 Application Master is hanging when the TaskImpl gets T_KILL event and completes attempts by the time
• is blocked by: MAPREDUCE-4748 Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED