[MAPREDUCE-2995] MR AM crashes when a container-launch hangs on a faulty NM

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Component/s: mrv2
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 13/Sep/11 09:49
• Updated: 10/Sep/24 07:57
• Resolved: 13/Sep/11 18:13

Description

AM tries to launch containers on a faulty node which blocks several/all of the StartContainer requests. Eventually, RM expires the container-allocations, informs the AM about container-expiry. But AM crashes with an INTERNAL_ERROR as the event is unexpected.

11/09/12 14:11:38 ERROR impl.TaskAttemptImpl: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_CONTAINER_COMPLETED at ASSIGNED
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:297)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:903)
        at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:127)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:543)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:536)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
        at java.lang.Thread.run(Thread.java:619)

Found this on a big cluster where karams was trying to get sort running.

Issue Links

• fixes: MAPREDUCE-7491 TaskAttemptImpl:Invalid event: TA_CONTAINER_COMPLETED at ASSIGNED
• incorporates: MAPREDUCE-2699 AM crashing because of invalid TA_CONTAINER_COMPLETED event