Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4598

Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In our cluster, I found that the container has some problems in state transition´╝îthis is my log

      2016-01-12 17:42:50,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000087 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
      2016-01-12 17:42:50,088 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Can't handle this event at current state: Current: 
      [CONTAINER_CLEANEDUP_AFTER_KILL], eventType: [RESOURCE_FAILED]
      org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL                                       
          at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)                                                                  
          at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)                                                                     
          at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)                                             
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1127)                                           
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)                                             
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1078)              
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1071)              
          at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)                                                                              
          at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)                                                                                 
          at java.lang.Thread.run(Thread.java:744)                                                                                                                        
      2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000094 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to null
      2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop   OPERATION=Container Finished - Killed   TARGET=ContainerImpl    
      RESULT=SUCCESS  APPID=application_1452588902899_0001    CONTAINERID=container_1452588902899_0001_01_000094                                                          
      2016-01-12 17:42:50,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1452588902899_0001_01_000094 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 
      

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9171 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9171/)
        YARN-4598. Invalid event: RESOURCE_FAILED at (jlowe: rev 46e5ea81e0e41646ba02391aea31481ead28aaf8)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9171 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9171/ ) YARN-4598 . Invalid event: RESOURCE_FAILED at (jlowe: rev 46e5ea81e0e41646ba02391aea31481ead28aaf8) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, tangshangwen! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.

        Show
        jlowe Jason Lowe added a comment - Thanks, tangshangwen ! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.
        Hide
        jlowe Jason Lowe added a comment -

        Test failure is unrelated. All Java tests passed but test-container-executor failed which is a known issue being tracked by YARN-4594.

        Committing this.

        Show
        jlowe Jason Lowe added a comment - Test failure is unrelated. All Java tests passed but test-container-executor failed which is a known issue being tracked by YARN-4594 . Committing this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 10m 32s trunk passed
        +1 compile 0m 53s trunk passed with JDK v1.8.0_66
        +1 compile 0m 34s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 15s trunk passed
        +1 javadoc 0m 31s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 45s the patch passed with JDK v1.8.0_66
        +1 javac 0m 45s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.7.0_91
        +1 javac 0m 33s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 32s the patch passed
        +1 javadoc 0m 33s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91
        -1 unit 10m 33s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 10m 1s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        43m 27s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782893/YARN-4598.1.patch
        JIRA Issue YARN-4598
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux baff2515a0f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d22c423
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10372/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10372/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10372/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Max memory used 77MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10372/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 32s trunk passed +1 compile 0m 53s trunk passed with JDK v1.8.0_66 +1 compile 0m 34s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 15s trunk passed +1 javadoc 0m 31s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 34s the patch passed +1 compile 0m 45s the patch passed with JDK v1.8.0_66 +1 javac 0m 45s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_91 +1 javac 0m 33s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91 -1 unit 10m 33s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. -1 unit 10m 1s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 43m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782893/YARN-4598.1.patch JIRA Issue YARN-4598 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux baff2515a0f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d22c423 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10372/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10372/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10372/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10372/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 pending Jenkins

        Show
        jlowe Jason Lowe added a comment - +1 pending Jenkins
        Hide
        tangshangwen tangshangwen added a comment -

        I submitted a patch

        Show
        tangshangwen tangshangwen added a comment - I submitted a patch
        Hide
        tangshangwen tangshangwen added a comment -

        I think we should add a transition , have any Suggestions?

           
         .addTransition(ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL,
                ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL,
                ContainerEventType.RESOURCE_FAILED)
        
        Show
        tangshangwen tangshangwen added a comment - I think we should add a transition , have any Suggestions? .addTransition(ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL, ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL, ContainerEventType.RESOURCE_FAILED)

          People

          • Assignee:
            tangshangwen tangshangwen
            Reporter:
            tangshangwen tangshangwen
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development