Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This breaks rolling upgrades because it changes the major version of the NM state store schema. Therefore when a new NM comes up on an old state store it crashes.

      The state store versions for this change have been updated in YARN-6798.
      Show
      This breaks rolling upgrades because it changes the major version of the NM state store schema. Therefore when a new NM comes up on an old state store it crashes. The state store versions for this change have been updated in YARN-6798 .

      Description

      YARN-1336 added the ability to restart NM without loosing any running containers. In a Federated YARN environment, there's additional state in the AMRMProxy to allow for spanning across multiple sub-clusters, so we need to enhance AMRMProxy to support work-preserving restart.

      1. YARN-6127.v1.patch
        68 kB
        Botong Huang
      2. YARN-6127.v2.patch
        75 kB
        Botong Huang
      3. YARN-6127.v3.patch
        72 kB
        Botong Huang
      4. YARN-6127.v4.patch
        72 kB
        Botong Huang
      5. YARN-6127-branch-2.v1.patch
        72 kB
        Botong Huang

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          Karthik Kambatla, you raise a valid concern. We addressed this as part of YARN-6798, so I'll try it to cherry-pick that back to branch-2 to make sure we don't break rolling upgrades.

          Show
          subru Subru Krishnan added a comment - Karthik Kambatla , you raise a valid concern. We addressed this as part of YARN-6798 , so I'll try it to cherry-pick that back to branch-2 to make sure we don't break rolling upgrades.
          Hide
          kasha Karthik Kambatla added a comment -

          Given this breaks rolling upgrade, are we sure we want to include in branch-2 and 2.9.0?

          Show
          kasha Karthik Kambatla added a comment - Given this breaks rolling upgrade, are we sure we want to include in branch-2 and 2.9.0?
          Hide
          botong Botong Huang added a comment -

          Thanks Arun Suresh and Subru Krishnan for the review!

          Show
          botong Botong Huang added a comment - Thanks Arun Suresh and Subru Krishnan for the review!
          Hide
          asuresh Arun Suresh added a comment -

          Not sure why Jenkins gave a compilation error.
          Committed to branch-2 as well. Thanks for the branch-2 patch Botong Huang. It works fine locally.

          Show
          asuresh Arun Suresh added a comment - Not sure why Jenkins gave a compilation error. Committed to branch-2 as well. Thanks for the branch-2 patch Botong Huang . It works fine locally.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 36s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 37s Maven dependency ordering for branch
          -1 mvninstall 0m 30s root in branch-2 failed.
          -1 compile 1m 19s hadoop-yarn-server in branch-2 failed with JDK v1.8.0_131.
          -1 compile 1m 14s hadoop-yarn-server in branch-2 failed with JDK v1.7.0_131.
          +1 checkstyle 0m 42s branch-2 passed
          +1 mvnsite 0m 57s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 1m 0s branch-2 passed
          +1 javadoc 0m 31s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 34s branch-2 passed with JDK v1.7.0_131
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 41s the patch passed
          -1 compile 0m 46s hadoop-yarn-server in the patch failed with JDK v1.8.0_131.
          -1 javac 0m 46s hadoop-yarn-server in the patch failed with JDK v1.8.0_131.
          -1 compile 0m 58s hadoop-yarn-server in the patch failed with JDK v1.7.0_131.
          -1 javac 0m 58s hadoop-yarn-server in the patch failed with JDK v1.7.0_131.
          +1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 198 unchanged - 4 fixed = 198 total (was 202)
          +1 mvnsite 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 0 new + 197 unchanged - 26 fixed = 197 total (was 223)
          +1 javadoc 0m 7s hadoop-yarn-server-tests in the patch passed with JDK v1.8.0_131.
          +1 javadoc 0m 29s the patch passed with JDK v1.7.0_131
          -1 unit 14m 17s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_131.
          -1 unit 7m 1s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_131.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          61m 50s



          Reason Tests
          JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestDiskFailures
          JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestDiskFailures



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue YARN-6127
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874122/YARN-6127-branch-2.v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9f92e8ede553 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 897d5a4
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_131.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_131.txt
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16224/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16224/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 36s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 37s Maven dependency ordering for branch -1 mvninstall 0m 30s root in branch-2 failed. -1 compile 1m 19s hadoop-yarn-server in branch-2 failed with JDK v1.8.0_131. -1 compile 1m 14s hadoop-yarn-server in branch-2 failed with JDK v1.7.0_131. +1 checkstyle 0m 42s branch-2 passed +1 mvnsite 0m 57s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 1m 0s branch-2 passed +1 javadoc 0m 31s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 34s branch-2 passed with JDK v1.7.0_131 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 41s the patch passed -1 compile 0m 46s hadoop-yarn-server in the patch failed with JDK v1.8.0_131. -1 javac 0m 46s hadoop-yarn-server in the patch failed with JDK v1.8.0_131. -1 compile 0m 58s hadoop-yarn-server in the patch failed with JDK v1.7.0_131. -1 javac 0m 58s hadoop-yarn-server in the patch failed with JDK v1.7.0_131. +1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 198 unchanged - 4 fixed = 198 total (was 202) +1 mvnsite 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 1m 3s the patch passed +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_131 with JDK v1.8.0_131 generated 0 new + 197 unchanged - 26 fixed = 197 total (was 223) +1 javadoc 0m 7s hadoop-yarn-server-tests in the patch passed with JDK v1.8.0_131. +1 javadoc 0m 29s the patch passed with JDK v1.7.0_131 -1 unit 14m 17s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_131. -1 unit 7m 1s hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 61m 50s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestDiskFailures JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestDiskFailures Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6127 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874122/YARN-6127-branch-2.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9f92e8ede553 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 897d5a4 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.8.0_131.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdk1.7.0_131.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_131.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16224/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16224/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/16224/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Re-opening to test branch-2 patch

          Show
          asuresh Arun Suresh added a comment - Re-opening to test branch-2 patch
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11908/)
          YARN-6127. Add support for work preserving NM restart when AMRMProxy is (arun suresh: rev 49aa60e50d20f8c18ed6f00fa8966244536fe7da)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyTokenSecretManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11908/ ) YARN-6127 . Add support for work preserving NM restart when AMRMProxy is (arun suresh: rev 49aa60e50d20f8c18ed6f00fa8966244536fe7da) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyTokenSecretManager.java
          Hide
          asuresh Arun Suresh added a comment -

          Committed this to trunk. Thanks Botong Huang

          Show
          asuresh Arun Suresh added a comment - Committed this to trunk. Thanks Botong Huang
          Hide
          botong Botong Huang added a comment -

          Thanks Arun Suresh! YARN-6730 created to follow up.

          Show
          botong Botong Huang added a comment - Thanks Arun Suresh ! YARN-6730 created to follow up.
          Hide
          asuresh Arun Suresh added a comment - - edited

          Will commit this shortly.
          One suggestion though - there are a lot of places where we check if the stateStore != null. This is true in the existing codebase too. Ideally, the stateStore should never be null and we should not have to perform so many defensive checks. Botong Huang, can you open a followup JIRA to fix this ?

          Show
          asuresh Arun Suresh added a comment - - edited Will commit this shortly. One suggestion though - there are a lot of places where we check if the stateStore != null. This is true in the existing codebase too. Ideally, the stateStore should never be null and we should not have to perform so many defensive checks. Botong Huang , can you open a followup JIRA to fix this ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 13m 55s trunk passed
          +1 compile 1m 53s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 0m 49s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 0m 31s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 1m 54s the patch passed
          +1 javac 1m 54s the patch passed
          +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 204 unchanged - 4 fixed = 204 total (was 208)
          +1 mvnsite 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227)
          +1 javadoc 0m 9s hadoop-yarn-server-tests in the patch passed.
          +1 unit 13m 13s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 6m 31s hadoop-yarn-server-tests in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          49m 23s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6127
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873975/YARN-6127.v4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5dd6d7602d6b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c22cf00
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16218/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16218/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16218/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16218/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 13m 55s trunk passed +1 compile 1m 53s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 49s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 31s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 1m 54s the patch passed +1 javac 1m 54s the patch passed +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 204 unchanged - 4 fixed = 204 total (was 208) +1 mvnsite 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 1m 1s the patch passed +1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227) +1 javadoc 0m 9s hadoop-yarn-server-tests in the patch passed. +1 unit 13m 13s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 6m 31s hadoop-yarn-server-tests in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 49m 23s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6127 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873975/YARN-6127.v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5dd6d7602d6b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c22cf00 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16218/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16218/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16218/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/16218/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          +1, pending jenkins..

          Show
          asuresh Arun Suresh added a comment - +1, pending jenkins..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 14m 38s trunk passed
          +1 compile 2m 9s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 0m 51s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 46s the patch passed
          +1 compile 2m 5s the patch passed
          +1 javac 2m 5s the patch passed
          +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 204 unchanged - 4 fixed = 204 total (was 208)
          +1 mvnsite 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 0m 57s the patch passed
          +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227)
          +1 javadoc 0m 8s hadoop-yarn-server-tests in the patch passed.
          +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 6m 35s hadoop-yarn-server-tests in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          51m 19s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6127
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873926/YARN-6127.v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 34a3f9210a94 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e806c6e
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16214/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16214/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16214/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 38s trunk passed +1 compile 2m 9s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 51s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 29s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 2m 5s the patch passed +1 javac 2m 5s the patch passed +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 204 unchanged - 4 fixed = 204 total (was 208) +1 mvnsite 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 0m 57s the patch passed +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227) +1 javadoc 0m 8s hadoop-yarn-server-tests in the patch passed. +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 6m 35s hadoop-yarn-server-tests in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 51m 19s Reason Tests Failed junit tests hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6127 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873926/YARN-6127.v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 34a3f9210a94 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e806c6e Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16214/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16214/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/16214/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          botong Botong Huang added a comment -

          Thanks Arun Suresh for the comments. v3 uploaded: NMSS main version incremented, added backward compatibility/robustness about the entries in NMSS. Can you please take another look? Thanks!

          Show
          botong Botong Huang added a comment - Thanks Arun Suresh for the comments. v3 uploaded: NMSS main version incremented, added backward compatibility/robustness about the entries in NMSS. Can you please take another look? Thanks!
          Hide
          asuresh Arun Suresh added a comment -

          Based on our offline discussions, I think I am OK with pushing the refactoring of the state to a separate JIRA - Since we have yet to fully hash out the implication of allowing Interceptors to access another Interceptor's stored state, since the recoveredDataMap will contain all interceptors' state for a particular application. We can fix that when we get to it.

          One minor nit though:

          • Instead of forcing the interceptor to call getRecoveredDataMap() from inside the recover() method, the map should probably be passed into the recover() method as an argument.
          Show
          asuresh Arun Suresh added a comment - Based on our offline discussions, I think I am OK with pushing the refactoring of the state to a separate JIRA - Since we have yet to fully hash out the implication of allowing Interceptors to access another Interceptor's stored state, since the recoveredDataMap will contain all interceptors' state for a particular application. We can fix that when we get to it. One minor nit though: Instead of forcing the interceptor to call getRecoveredDataMap() from inside the recover() method, the map should probably be passed into the recover() method as an argument.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the patch Botong Huang

          Couple of comments:

          • It looks like when an interceptor needs to persist state, it has to explicitly do an nmContext.getNMStateStore().storeAMRMProxyAppContextEntry() while after recovery, it must explicitly invoke the getRecoveredDataMap() to access the state. I feel it might be better to just expose an InterceptorState API/class that is available to the Interceptor via the context. This state object can then expose a get(key) and put(key, value) which would under the hood negotiate with the stateStore to store the state and retrieve all existing keys and values on recovery.
          • We should be incrementing the major version of the version Info. Also, I think we would need to do something similar to YARN-5547 to handle the AMRMPROXY_KEY_PREFIX to ensure that rollback does not bomb.
          Show
          asuresh Arun Suresh added a comment - Thanks for the patch Botong Huang Couple of comments: It looks like when an interceptor needs to persist state, it has to explicitly do an nmContext.getNMStateStore().storeAMRMProxyAppContextEntry() while after recovery, it must explicitly invoke the getRecoveredDataMap() to access the state. I feel it might be better to just expose an InterceptorState API/class that is available to the Interceptor via the context. This state object can then expose a get(key) and put(key, value) which would under the hood negotiate with the stateStore to store the state and retrieve all existing keys and values on recovery. We should be incrementing the major version of the version Info. Also, I think we would need to do something similar to YARN-5547 to handle the AMRMPROXY_KEY_PREFIX to ensure that rollback does not bomb.
          Hide
          botong Botong Huang added a comment -

          Unit test failures are not related:
          TestMiniYarnClusterNodeUtilization open in https://issues.apache.org/jira/browse/YARN-4453
          TestContainerManagerSecurity being discussed in https://issues.apache.org/jira/browse/YARN-5655

          Show
          botong Botong Huang added a comment - Unit test failures are not related: TestMiniYarnClusterNodeUtilization open in https://issues.apache.org/jira/browse/YARN-4453 TestContainerManagerSecurity being discussed in https://issues.apache.org/jira/browse/YARN-5655
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 37s Maven dependency ordering for branch
          +1 mvninstall 13m 11s trunk passed
          +1 compile 1m 38s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 47s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 40s the patch passed
          +1 compile 1m 38s the patch passed
          +1 javac 1m 38s the patch passed
          +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 203 unchanged - 4 fixed = 203 total (was 207)
          +1 mvnsite 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 0m 49s the patch passed
          +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227)
          +1 javadoc 0m 9s hadoop-yarn-server-tests in the patch passed.
          +1 unit 12m 43s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 6m 27s hadoop-yarn-server-tests in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          47m 39s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6127
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872129/YARN-6127.v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 38bc32e6fb4d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a062374
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16159/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16159/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16159/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16159/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 37s Maven dependency ordering for branch +1 mvninstall 13m 11s trunk passed +1 compile 1m 38s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 29s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 40s the patch passed +1 compile 1m 38s the patch passed +1 javac 1m 38s the patch passed +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new + 203 unchanged - 4 fixed = 203 total (was 207) +1 mvnsite 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 0m 49s the patch passed +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 201 unchanged - 26 fixed = 201 total (was 227) +1 javadoc 0m 9s hadoop-yarn-server-tests in the patch passed. +1 unit 12m 43s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 6m 27s hadoop-yarn-server-tests in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 47m 39s Reason Tests Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6127 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872129/YARN-6127.v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 38bc32e6fb4d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a062374 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16159/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16159/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16159/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/16159/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 13m 22s trunk passed
          +1 compile 1m 37s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 45s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 40s the patch passed
          +1 compile 1m 53s the patch passed
          +1 javac 1m 53s the patch passed
          -0 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 8 new + 205 unchanged - 2 fixed = 213 total (was 207)
          +1 mvnsite 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
          +1 findbugs 1m 1s the patch passed
          -1 javadoc 0m 19s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 12m 59s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 6m 28s hadoop-yarn-server-tests in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          48m 34s



          Reason Tests
          Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
            hadoop.yarn.server.TestContainerManagerSecurity



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6127
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871990/YARN-6127.v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 19dadfb83102 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5672ae7
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16149/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16149/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 13m 22s trunk passed +1 compile 1m 37s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 45s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 29s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 40s the patch passed +1 compile 1m 53s the patch passed +1 javac 1m 53s the patch passed -0 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 8 new + 205 unchanged - 2 fixed = 213 total (was 207) +1 mvnsite 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests +1 findbugs 1m 1s the patch passed -1 javadoc 0m 19s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 12m 59s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 6m 28s hadoop-yarn-server-tests in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 48m 34s Reason Tests Failed junit tests hadoop.yarn.server.TestMiniYarnClusterNodeUtilization   hadoop.yarn.server.TestContainerManagerSecurity Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6127 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871990/YARN-6127.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 19dadfb83102 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5672ae7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16149/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16149/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/16149/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              botong Botong Huang
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development