[YARN-4598] Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
• Component/s: nodemanager
• Target Version/s: 2.7.3, 2.6.4
• Hadoop Flags: Reviewed
• Assignee: Shangwen Tang
• Reporter: Shangwen Tang
• Created: 15/Jan/16 06:48
• Updated: 06/Jan/17 07:34
• Resolved: 22/Jan/16 22:00

Description

In our cluster, I found that the container has some problems in state transition，this is my log

2016-01-12 17:42:50,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000087 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2016-01-12 17:42:50,088 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Can't handle this event at current state: Current: 
[CONTAINER_CLEANEDUP_AFTER_KILL], eventType: [RESOURCE_FAILED]
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL                                       
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)                                                                  
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)                                                                     
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)                                             
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1127)                                           
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)                                             
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1078)              
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1071)              
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)                                                                              
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)                                                                                 
    at java.lang.Thread.run(Thread.java:744)                                                                                                                        
2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000094 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to null
2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop   OPERATION=Container Finished - Killed   TARGET=ContainerImpl    
RESULT=SUCCESS  APPID=application_1452588902899_0001    CONTAINERID=container_1452588902899_0001_01_000094                                                          
2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000094 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE