Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4831

Recovered containers will be killed after NM stateful restart

    Details

    • Hadoop Flags:
      Reviewed

      Description

      2016-03-04 19:43:48,130 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1456335621285_0040_01_000066 transitioned from NEW to DONE
      2016-03-04 19:43:48,130 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=henkins-service	OPERATION=Container Finished - Killed	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1456335621285_0040
      

        Activity

        Hide
        l201514 Siqi Li added a comment -

        When NM does a stateful restart, the ContainerManagerImpl will try to recover applications, and containers, and then send out ApplicationFinishEvent to apps that in appsState.getFinishedApplications().

        The ApplicationFinishEvent could result in newly recovered containers to transit from NEW to DONE with a KillOnNewTransition.
        We could add an additional check in KillOnNewTransition to avoid killing completed containers.

        Show
        l201514 Siqi Li added a comment - When NM does a stateful restart, the ContainerManagerImpl will try to recover applications, and containers, and then send out ApplicationFinishEvent to apps that in appsState.getFinishedApplications(). The ApplicationFinishEvent could result in newly recovered containers to transit from NEW to DONE with a KillOnNewTransition. We could add an additional check in KillOnNewTransition to avoid killing completed containers.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 3s trunk passed
        +1 compile 0m 23s trunk passed with JDK v1.8.0_74
        +1 compile 0m 27s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 53s trunk passed
        +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 21s the patch passed with JDK v1.8.0_74
        +1 javac 0m 21s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_95
        +1 javac 0m 25s the patch passed
        -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 91 unchanged - 0 fixed = 92 total (was 91)
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 3s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95
        +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 50s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        34m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793824/YARN-4831.v1.patch
        JIRA Issue YARN-4831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e771231775da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ae14e5d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10801/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10801/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10801/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 3s trunk passed +1 compile 0m 23s trunk passed with JDK v1.8.0_74 +1 compile 0m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 25s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_74 +1 javac 0m 21s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_95 +1 javac 0m 25s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 91 unchanged - 0 fixed = 92 total (was 91) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95 +1 unit 9m 11s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 50s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 34m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793824/YARN-4831.v1.patch JIRA Issue YARN-4831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e771231775da 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ae14e5d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10801/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10801/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10801/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Sorry for the long delay in reviewing.

        +1 lgtm. Will commit this pending a fresh Jenkins run.

        Show
        jlowe Jason Lowe added a comment - Sorry for the long delay in reviewing. +1 lgtm. Will commit this pending a fresh Jenkins run.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 37s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 49s trunk passed
        +1 javadoc 0m 18s trunk passed
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        -0 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 90 unchanged - 0 fixed = 91 total (was 90)
        +1 mvnsite 0m 30s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 58s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 15m 23s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        30m 28s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-4831
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793824/YARN-4831.v1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 10e0429632a5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac35ee9
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13559/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13559/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13559/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13559/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 37s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -0 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 90 unchanged - 0 fixed = 91 total (was 90) +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 15m 23s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 30m 28s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-4831 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793824/YARN-4831.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 10e0429632a5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac35ee9 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13559/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13559/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13559/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13559/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        The unit test failure is unrelated, and the test passes for me locally with the patch applied. Committing this.

        Show
        jlowe Jason Lowe added a comment - The unit test failure is unrelated, and the test passes for me locally with the patch applied. Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Siqi! I committed this to trunk, branch-2, and branch-2.8.

        Show
        jlowe Jason Lowe added a comment - Thanks, Siqi! I committed this to trunk, branch-2, and branch-2.8.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10710 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10710/)
        YARN-4831. Recovered containers will be killed after NM stateful (jlowe: rev 7e3c327d316b33d6a09bfd4e65e7e5384943bb1d)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10710 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10710/ ) YARN-4831 . Recovered containers will be killed after NM stateful (jlowe: rev 7e3c327d316b33d6a09bfd4e65e7e5384943bb1d) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java

          People

          • Assignee:
            l201514 Siqi Li
            Reporter:
            l201514 Siqi Li
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development