Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3542

Re-factor support for CPU as a resource using the new ResourceHandler mechanism

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In YARN-3443 , a new ResourceHandler mechanism was added which enabled easier addition of new resource types in the nodemanager (this was used for network as a resource - See YARN-2140 ). We should refactor the existing CPU implementation ( LinuxContainerExecutor/CgroupsLCEResourcesHandler ) using the new ResourceHandler mechanism.

      1. YARN-3542.001.patch
        28 kB
        Varun Vasudev
      2. YARN-3542.002.patch
        29 kB
        Varun Vasudev
      3. YARN-3542.003.patch
        32 kB
        Varun Vasudev
      4. YARN-3542.004.patch
        41 kB
        Varun Vasudev
      5. YARN-3542.005.patch
        41 kB
        Varun Vasudev
      6. YARN-3542.006.patch
        42 kB
        Varun Vasudev
      7. YARN-3542.007.patch
        42 kB
        Varun Vasudev
      8. YARN-3542.008.patch
        44 kB
        Varun Vasudev

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9184 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9184/)
          YARN-3542. Refactored existing CPU cgroups support to use the newer and (vinodkv: rev 2085e60a9655b59aa2ba8917acdc511ab71ff6e4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/DefaultLCEResourcesHandler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsCpuResourceHandlerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CpuResourceHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/LCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9184 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9184/ ) YARN-3542 . Refactored existing CPU cgroups support to use the newer and (vinodkv: rev 2085e60a9655b59aa2ba8917acdc511ab71ff6e4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/DefaultLCEResourcesHandler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsCpuResourceHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CpuResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/LCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk and branch-2. Thanks Varun!

          Tx for all the reviews, Sid!

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Committed this to trunk and branch-2. Thanks Varun! Tx for all the reviews, Sid!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Your proposal to defer the removal for later is reasonable.

          The latest patch looks good to me. +1. Checking this in.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Your proposal to defer the removal for later is reasonable. The latest patch looks good to me. +1. Checking this in.
          Hide
          vvasudev Varun Vasudev added a comment -

          The increase in javac warnings is due to the deprecation of the LCEResourcesHandler interface. The test failures are unrelated to this patch - they're in the container executor test binary.

          Show
          vvasudev Varun Vasudev added a comment - The increase in javac warnings is due to the deprecation of the LCEResourcesHandler interface. The test failures are unrelated to this patch - they're in the container executor test binary.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 8m 19s trunk passed
          +1 compile 2m 0s trunk passed with JDK v1.8.0_66
          +1 compile 2m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 2m 28s trunk passed
          +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 20s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 50s the patch passed
          +1 compile 1m 58s the patch passed with JDK v1.8.0_66
          -1 javac 3m 5s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 13 new + 9 unchanged - 0 fixed = 22 total (was 9)
          +1 javac 1m 58s the patch passed
          +1 compile 2m 20s the patch passed with JDK v1.7.0_91
          -1 javac 5m 25s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 15 new + 10 unchanged - 0 fixed = 25 total (was 10)
          +1 javac 2m 20s the patch passed
          -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn: patch generated 5 new + 259 unchanged - 9 fixed = 264 total (was 268)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 48s the patch passed
          +1 javadoc 0m 58s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 21s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          -1 unit 8m 45s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          -1 unit 9m 13s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          56m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783825/YARN-3542.008.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 235a23d38968 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0bae506
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10368/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10368/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 8m 19s trunk passed +1 compile 2m 0s trunk passed with JDK v1.8.0_66 +1 compile 2m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 2m 28s trunk passed +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 20s trunk passed with JDK v1.7.0_91 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 1m 58s the patch passed with JDK v1.8.0_66 -1 javac 3m 5s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 13 new + 9 unchanged - 0 fixed = 22 total (was 9) +1 javac 1m 58s the patch passed +1 compile 2m 20s the patch passed with JDK v1.7.0_91 -1 javac 5m 25s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 15 new + 10 unchanged - 0 fixed = 25 total (was 10) +1 javac 2m 20s the patch passed -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn: patch generated 5 new + 259 unchanged - 9 fixed = 264 total (was 268) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 48s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 21s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. -1 unit 8m 45s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. -1 unit 9m 13s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 56m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783825/YARN-3542.008.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 235a23d38968 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0bae506 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10368/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10368/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10368/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new version of the patch which also deprecates the DefaultResourceHandler and LCEResourceHandler interface which is something Vinod Kumar Vavilapalli wanted as part of his review.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new version of the patch which also deprecates the DefaultResourceHandler and LCEResourceHandler interface which is something Vinod Kumar Vavilapalli wanted as part of his review.
          Hide
          vvasudev Varun Vasudev added a comment -

          This effectively means the old code is not used anymore, and that the new code is stable.

          And it seems like we are stating that

          [..], there should be no issue hooking into the new handler using the old configuration mechanism.?

          Given this and the fact that we are internally overriding to use the new handlers, is there a reason for keeping the old code at all? Also if we are using the new handler code internally anyways, we can proceed with the deprecation (or better deletion) of LCEResourcesHandler interface, DefaultLCEResourcesHandler etc?

          I think there are two issues here -
          1) should we keep the CgroupsLCEResourcesHandler code and
          2) should we do away with the resource-handler code altogether

          The answer to (1) is that it is not required but I think it's useful to keep around for some time for two reasons -
          1. As documentation/reference - in case some discrepancy is seen in the new implementation
          2. In the off-chance that someone has inherited from it. Unfortunately the resource-handler classes are public via config but are not annotated as such.

          The answer to (2) is no - the resource handler interface is public and we shouldn't break it. Deprecating it is fine once we provide an alternative(which we currently don't have) or decide we will not support it any longer. Either way that decision should be a seperate JIRA, not this one.

          Show
          vvasudev Varun Vasudev added a comment - This effectively means the old code is not used anymore, and that the new code is stable. And it seems like we are stating that [..] , there should be no issue hooking into the new handler using the old configuration mechanism.? Given this and the fact that we are internally overriding to use the new handlers, is there a reason for keeping the old code at all? Also if we are using the new handler code internally anyways, we can proceed with the deprecation (or better deletion) of LCEResourcesHandler interface, DefaultLCEResourcesHandler etc? I think there are two issues here - 1) should we keep the CgroupsLCEResourcesHandler code and 2) should we do away with the resource-handler code altogether The answer to (1) is that it is not required but I think it's useful to keep around for some time for two reasons - 1. As documentation/reference - in case some discrepancy is seen in the new implementation 2. In the off-chance that someone has inherited from it. Unfortunately the resource-handler classes are public via config but are not annotated as such. The answer to (2) is no - the resource handler interface is public and we shouldn't break it. Deprecating it is fine once we provide an alternative(which we currently don't have) or decide we will not support it any longer. Either way that decision should be a seperate JIRA, not this one.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Okay, clearly most of my comments were rooted on the following question

          This effectively means the old code is not used anymore, and that the new code is stable.

          And it seems like we are stating that

          [..], there should be no issue hooking into the new handler using the old configuration mechanism.?

          Given this and the fact that we are internally overriding to use the new handlers, is there a reason for keeping the old code at all? Also if we are using the new handler code internally anyways, we can proceed with the deprecation (or better deletion) of LCEResourcesHandler interface, DefaultLCEResourcesHandler etc?

          When user sets the old CgroupsLCEResourcesHandler, you are internally resetting it to DefaultLCEResourcesHandler(inside LinuxContainerExecutor) and using that as a control to stop using the older handler.

          Instead of doing this implicitly, how about we completely remove LinuxContainerExecutor.resourcesHandler etc as they don't perform any real function anymore.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Okay, clearly most of my comments were rooted on the following question This effectively means the old code is not used anymore, and that the new code is stable. And it seems like we are stating that [..] , there should be no issue hooking into the new handler using the old configuration mechanism.? Given this and the fact that we are internally overriding to use the new handlers, is there a reason for keeping the old code at all? Also if we are using the new handler code internally anyways, we can proceed with the deprecation (or better deletion) of LCEResourcesHandler interface, DefaultLCEResourcesHandler etc? When user sets the old CgroupsLCEResourcesHandler, you are internally resetting it to DefaultLCEResourcesHandler(inside LinuxContainerExecutor) and using that as a control to stop using the older handler. Instead of doing this implicitly, how about we completely remove LinuxContainerExecutor.resourcesHandler etc as they don't perform any real function anymore.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 25s trunk passed
          +1 compile 2m 0s trunk passed with JDK v1.8.0_66
          +1 compile 2m 15s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 24s trunk passed
          +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 56s the patch passed
          +1 compile 2m 5s the patch passed with JDK v1.8.0_66
          -1 javac 8m 17s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 9, now 10).
          +1 javac 2m 5s the patch passed
          +1 compile 2m 18s the patch passed with JDK v1.7.0_91
          -1 javac 10m 35s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 10, now 12).
          +1 javac 2m 18s the patch passed
          -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 269, now 265).
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 44s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 15s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 36s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          57m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779078/YARN-3542.007.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3b4dee099c08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 70d6f20
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10074/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10074/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 25s trunk passed +1 compile 2m 0s trunk passed with JDK v1.8.0_66 +1 compile 2m 15s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 24s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 56s the patch passed +1 compile 2m 5s the patch passed with JDK v1.8.0_66 -1 javac 8m 17s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 9, now 10). +1 javac 2m 5s the patch passed +1 compile 2m 18s the patch passed with JDK v1.7.0_91 -1 javac 10m 35s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 10, now 12). +1 javac 2m 18s the patch passed -1 checkstyle 0m 29s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 269, now 265). +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 44s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 15s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 9m 21s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 36s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 57m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779078/YARN-3542.007.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3b4dee099c08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 70d6f20 findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10074/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10074/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10074/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch that'll apply on trunk. Just a note on the configurations - all the existing configurations for CPU isolation from CGroupsLCEResourceHandler carry over without any change. My expectation is that we'll continue to use them going forward. The only issue we haven't quite figured out is the configs for enabling the resource handlers.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch that'll apply on trunk. Just a note on the configurations - all the existing configurations for CPU isolation from CGroupsLCEResourceHandler carry over without any change. My expectation is that we'll continue to use them going forward. The only issue we haven't quite figured out is the configs for enabling the resource handlers.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Hi Vinod,

          A few comments inline regarding your review comments.

          Thanks,
          -Sidharta

          When user sets the old CgroupsLCEResourcesHandler, you are internally resetting it to DefaultLCEResourcesHandler(inside LinuxContainerExecutor) and using that as a control to stop using the older handler. This effectively means the old code is not used anymore, and that the new code is stable. This doesn't map well with the earlier decision to not document the new handlers yet.

          Thats a good point. However, the handlers themselves aren’t the issue IMO, but rather the configuration which might get out of hand as we add more resource handlers. This is especially true in light of resource profiles being worked in in YARN-3926 - which might require some changes to how the resource handlers are configured. However, there should be no issue hooking into the new handler using the old configuration mechanism.

          Lot more dedup is possible between the new hierarchy and the older hierarchy

          I don’t think we should dedup this code. There is no reason to hook into the new code/handlers for either of the two scenarios being discussed here : 1) deprecate the old handler in which case there shouldn’t be any reason to make make major changes to it. 2) not deprecate the old handler because the new handler might not be stable - in which case it doesn’t make sense to hook into the new handler/code yet.

          Further, specific constants like CPU_PERIOD_US perhaps belong better to the specific implementations likeCGroupsCpuResourceHandlerImpl instead of the root CpuResourceHandler.

          I am assuming you meant this : ‘the root CGroupsHandler’ not ‘the root CpuResourceHandler’. Arguments can be made both ways here : CGroupsHandler already has enums and constants that are used across multiple resource handler implementations. Some of these cannot be moved out (e.g the enum, tasks etc) . Moving out some of these to individual handlers is of limited use and makes it hard to get an quick overview of all the cgroups subsystems/parameters in use among the various resource handlers. It also creates problems if new handlers for the same subsystem are created which require using the same cgroup parameters. Cleaning this up fully would require more extensive refactoring - individual classes for various cgroups subsystems etc., This is not necessary, IMO.

          We should also deprecate the old LCEResourcesHandler interface, DefaultLCEResourcesHandler etc. That said, we shouldn't deprecate them or CgroupsLCEResourcesHandler before we make the newer mechanism public and usable. So may be we should fork off all this deprecation to another JIRA and only get to it after we publicly document the new mechanism for stable usage.

          Yeah, thats a good point again. The new handler itself isn’t the issue - the new configuration could be. Like I said before, though, I think there should be no problem using the new handler if the old configuration mechanism is in use.

          Show
          sidharta-s Sidharta Seethana added a comment - Hi Vinod, A few comments inline regarding your review comments. Thanks, -Sidharta When user sets the old CgroupsLCEResourcesHandler, you are internally resetting it to DefaultLCEResourcesHandler(inside LinuxContainerExecutor) and using that as a control to stop using the older handler. This effectively means the old code is not used anymore, and that the new code is stable. This doesn't map well with the earlier decision to not document the new handlers yet. Thats a good point. However, the handlers themselves aren’t the issue IMO, but rather the configuration which might get out of hand as we add more resource handlers. This is especially true in light of resource profiles being worked in in YARN-3926 - which might require some changes to how the resource handlers are configured. However, there should be no issue hooking into the new handler using the old configuration mechanism. Lot more dedup is possible between the new hierarchy and the older hierarchy I don’t think we should dedup this code. There is no reason to hook into the new code/handlers for either of the two scenarios being discussed here : 1) deprecate the old handler in which case there shouldn’t be any reason to make make major changes to it. 2) not deprecate the old handler because the new handler might not be stable - in which case it doesn’t make sense to hook into the new handler/code yet. Further, specific constants like CPU_PERIOD_US perhaps belong better to the specific implementations likeCGroupsCpuResourceHandlerImpl instead of the root CpuResourceHandler. I am assuming you meant this : ‘the root CGroupsHandler’ not ‘the root CpuResourceHandler’. Arguments can be made both ways here : CGroupsHandler already has enums and constants that are used across multiple resource handler implementations. Some of these cannot be moved out (e.g the enum, tasks etc) . Moving out some of these to individual handlers is of limited use and makes it hard to get an quick overview of all the cgroups subsystems/parameters in use among the various resource handlers. It also creates problems if new handlers for the same subsystem are created which require using the same cgroup parameters. Cleaning this up fully would require more extensive refactoring - individual classes for various cgroups subsystems etc., This is not necessary, IMO. We should also deprecate the old LCEResourcesHandler interface, DefaultLCEResourcesHandler etc. That said, we shouldn't deprecate them or CgroupsLCEResourcesHandler before we make the newer mechanism public and usable. So may be we should fork off all this deprecation to another JIRA and only get to it after we publicly document the new mechanism for stable usage. Yeah, thats a good point again. The new handler itself isn’t the issue - the new configuration could be. Like I said before, though, I think there should be no problem using the new handler if the old configuration mechanism is in use.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          First up, the patch doesn't apply anymore after YARN-1856.

          Reviewing the patch on top of a commit pre YARN-1856, some comments:

          • When user sets the old CgroupsLCEResourcesHandler, you are internally resetting it to DefaultLCEResourcesHandler (inside LinuxContainerExecutor) and using that as a control to stop using the older handler. This effectively means the old code is not used anymore, and that the new code is stable. This doesn't map well with the earlier decision to not document the new handlers yet.
          • Lot more dedup is possible between the new hierarchy and the older hierarchy
            • We can duplicate definitions of constants like MTAB_FILE, CPU_PERIOD_US in the old CgroupsLCEResourcesHandler in favor of routing them to the new ones in CGroupsHandler?
            • Further, specific constants like CPU_PERIOD_US perhaps belong better to the specific implementations like CGroupsCpuResourceHandlerImpl instead of the root CpuResourceHandler.
            • The following pieces of code are almost identical and can be deduped:
              • CgroupsLCEResourcesHandler.updateCgroup and CgroupsHandlerImpl's updateCGroupParam
              • CgroupsLCEResourcesHandler.deleteCgroup and CGroupsHandlerImpl.deleteCGroup
              • CgroupsLCEResourcesHandler.parseMtab and CGroupsHandlerImpl.parseMtab
          • We should also deprecate the old LCEResourcesHandler interface, DefaultLCEResourcesHandler etc. That said, we shouldn't deprecate them or CgroupsLCEResourcesHandler before we make the newer mechanism public and usable. So may be we should fork off all this deprecation to another JIRA and only get to it after we publicly document the new mechanism for stable usage.
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - First up, the patch doesn't apply anymore after YARN-1856 . Reviewing the patch on top of a commit pre YARN-1856 , some comments: When user sets the old CgroupsLCEResourcesHandler , you are internally resetting it to DefaultLCEResourcesHandler (inside LinuxContainerExecutor ) and using that as a control to stop using the older handler. This effectively means the old code is not used anymore, and that the new code is stable. This doesn't map well with the earlier decision to not document the new handlers yet. Lot more dedup is possible between the new hierarchy and the older hierarchy We can duplicate definitions of constants like MTAB_FILE, CPU_PERIOD_US in the old CgroupsLCEResourcesHandler in favor of routing them to the new ones in CGroupsHandler ? Further, specific constants like CPU_PERIOD_US perhaps belong better to the specific implementations like CGroupsCpuResourceHandlerImpl instead of the root CpuResourceHandler . The following pieces of code are almost identical and can be deduped: CgroupsLCEResourcesHandler.updateCgroup and CgroupsHandlerImpl's updateCGroupParam CgroupsLCEResourcesHandler.deleteCgroup and CGroupsHandlerImpl.deleteCGroup CgroupsLCEResourcesHandler.parseMtab and CGroupsHandlerImpl.parseMtab We should also deprecate the old LCEResourcesHandler interface, DefaultLCEResourcesHandler etc. That said, we shouldn't deprecate them or CgroupsLCEResourcesHandler before we make the newer mechanism public and usable. So may be we should fork off all this deprecation to another JIRA and only get to it after we publicly document the new mechanism for stable usage.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Thanks, Varun Vasudev. +1 on the latest patch from me.

          Show
          sidharta-s Sidharta Seethana added a comment - Thanks, Varun Vasudev . +1 on the latest patch from me.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 27s trunk passed
          +1 compile 2m 6s trunk passed with JDK v1.8.0_66
          +1 compile 2m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 25s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 55s the patch passed
          +1 compile 2m 9s the patch passed with JDK v1.8.0_66
          -1 javac 8m 20s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 9, now 10).
          +1 javac 2m 9s the patch passed
          +1 compile 2m 17s the patch passed with JDK v1.7.0_91
          -1 javac 10m 38s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 10, now 12).
          +1 javac 2m 17s the patch passed
          -1 checkstyle 0m 28s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 268, now 264).
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 43s the patch passed
          +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          56m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777937/YARN-3542.006.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f885fab9ff8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7092d47
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10003/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10003/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 27s trunk passed +1 compile 2m 6s trunk passed with JDK v1.8.0_66 +1 compile 2m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 25s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 55s the patch passed +1 compile 2m 9s the patch passed with JDK v1.8.0_66 -1 javac 8m 20s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 9, now 10). +1 javac 2m 9s the patch passed +1 compile 2m 17s the patch passed with JDK v1.7.0_91 -1 javac 10m 38s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 2 new issues (was 10, now 12). +1 javac 2m 17s the patch passed -1 checkstyle 0m 28s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 268, now 264). +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 43s the patch passed +1 javadoc 1m 0s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91 +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 9m 0s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 9m 22s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 56m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777937/YARN-3542.006.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f885fab9ff8e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7092d47 findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10003/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10003/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10003/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Adding a @Deprecated to TestCgroupsLCEResourcesHandler would remove the deprecated warning use - this test class should be deprecated anyway?

          Yep. Fixed.

          Some checkstyle comments need looking into (line lengths for new code, mostly)

          Fixed as many as I could - the rest are package lines and one config variable.

          Show
          vvasudev Varun Vasudev added a comment - Adding a @Deprecated to TestCgroupsLCEResourcesHandler would remove the deprecated warning use - this test class should be deprecated anyway? Yep. Fixed. Some checkstyle comments need looking into (line lengths for new code, mostly) Fixed as many as I could - the rest are package lines and one config variable.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Thanks for the updated patch, Varun Vasudev. A couple of additional things :

          • Adding a @Deprecated to TestCgroupsLCEResourcesHandler would remove the deprecated warning use - this test class should be deprecated anyway?
          • Some checkstyle comments need looking into (line lengths for new code, mostly)
          Show
          sidharta-s Sidharta Seethana added a comment - Thanks for the updated patch, Varun Vasudev . A couple of additional things : Adding a @Deprecated to TestCgroupsLCEResourcesHandler would remove the deprecated warning use - this test class should be deprecated anyway? Some checkstyle comments need looking into (line lengths for new code, mostly)
          Hide
          vvasudev Varun Vasudev added a comment -

          The javac warning increase is due to the deprecation of CgroupsLCEResourcesHandler and its use in TestCgroupsLCEResourcesHandler.

          Show
          vvasudev Varun Vasudev added a comment - The javac warning increase is due to the deprecation of CgroupsLCEResourcesHandler and its use in TestCgroupsLCEResourcesHandler.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 32s trunk passed
          +1 compile 1m 51s trunk passed with JDK v1.8.0_66
          +1 compile 2m 6s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 2m 16s trunk passed
          +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 5s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 54s the patch passed
          +1 compile 1m 51s the patch passed with JDK v1.8.0_66
          -1 javac 7m 39s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 8 new issues (was 9, now 17).
          +1 javac 1m 51s the patch passed
          +1 compile 2m 8s the patch passed with JDK v1.7.0_91
          -1 javac 9m 47s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 8 new issues (was 10, now 18).
          +1 javac 2m 8s the patch passed
          -1 checkstyle 0m 28s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 267, now 264).
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 33s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 9s the patch passed with JDK v1.7.0_91
          +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          53m 5s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777774/YARN-3542.005.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c15ca68c695d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0c3a53e
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9987/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 75MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9987/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 32s trunk passed +1 compile 1m 51s trunk passed with JDK v1.8.0_66 +1 compile 2m 6s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 5s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 54s the patch passed +1 compile 1m 51s the patch passed with JDK v1.8.0_66 -1 javac 7m 39s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 8 new issues (was 9, now 17). +1 javac 1m 51s the patch passed +1 compile 2m 8s the patch passed with JDK v1.7.0_91 -1 javac 9m 47s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 8 new issues (was 10, now 18). +1 javac 2m 8s the patch passed -1 checkstyle 0m 28s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 267, now 264). +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 33s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 9s the patch passed with JDK v1.7.0_91 +1 unit 0m 23s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 8m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 53m 5s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777774/YARN-3542.005.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c15ca68c695d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0c3a53e findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9987/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9987/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9987/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a patch to fix a failing test.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a patch to fix a failing test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 59s trunk passed
          +1 compile 2m 1s trunk passed with JDK v1.8.0_66
          +1 compile 2m 15s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 22s trunk passed
          +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 12s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 55s the patch passed
          +1 compile 2m 2s the patch passed with JDK v1.8.0_66
          -1 javac 8m 7s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 8 new issues (was 9, now 17).
          +1 javac 2m 2s the patch passed
          +1 compile 2m 17s the patch passed with JDK v1.7.0_91
          -1 javac 10m 24s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 8 new issues (was 10, now 18).
          +1 javac 2m 17s the patch passed
          -1 checkstyle 0m 30s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 268, now 265).
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 45s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 19s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          -1 unit 9m 33s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          -1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          56m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsCpuResourceHandlerImpl
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsCpuResourceHandlerImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777759/YARN-3542.004.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 92024f0b813c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0c3a53e
          findbugs v3.0.0
          javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt
          javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9984/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0 http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9984/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 2m 1s trunk passed with JDK v1.8.0_66 +1 compile 2m 15s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 22s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 12s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 55s the patch passed +1 compile 2m 2s the patch passed with JDK v1.8.0_66 -1 javac 8m 7s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK v1.8.0_66 generated 8 new issues (was 9, now 17). +1 javac 2m 2s the patch passed +1 compile 2m 17s the patch passed with JDK v1.7.0_91 -1 javac 10m 24s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91 with JDK v1.7.0_91 generated 8 new issues (was 10, now 18). +1 javac 2m 17s the patch passed -1 checkstyle 0m 30s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 268, now 265). +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 45s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 19s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. -1 unit 9m 33s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. -1 unit 9m 20s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 56m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsCpuResourceHandlerImpl JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsCpuResourceHandlerImpl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777759/YARN-3542.004.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 92024f0b813c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0c3a53e findbugs v3.0.0 javac hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66.txt javac hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9984/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9984/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9984/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address Sidharta Seethana's comments.

          unused imports in CGroupsCpuResourceHandlerImpl

          Fixed.

          Move these to CGroupsHandler ?

          Fixed.

          About how shares/cfs_period_us/cfs_quota_us are used : additional comments/documentation and examples (as unit tests?) would be useful. It took me a while to trace through the code using some examples.

          Added comments on how they're used. I also added a bunch of new unit tests.

          Since the class has been marked deprecated, is it necessary to make the rest of the changes that are included ?

          I think the class should exist as is for now - the unit tests for the class would fail without the changes. We can decide later whether to remove it entirely.

          IMO, all references to LCEResourcesHandler (and sub-classes) should be removed from LinuxContainerExecutor.

          The LCEResourcesHandler is a public API(exposed via yarn-site.xml) - users may have implemented their own. Removing it would mean breaking backwards compatibility.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address Sidharta Seethana 's comments. unused imports in CGroupsCpuResourceHandlerImpl Fixed. Move these to CGroupsHandler ? Fixed. About how shares/cfs_period_us/cfs_quota_us are used : additional comments/documentation and examples (as unit tests?) would be useful. It took me a while to trace through the code using some examples. Added comments on how they're used. I also added a bunch of new unit tests. Since the class has been marked deprecated, is it necessary to make the rest of the changes that are included ? I think the class should exist as is for now - the unit tests for the class would fail without the changes. We can decide later whether to remove it entirely. IMO, all references to LCEResourcesHandler (and sub-classes) should be removed from LinuxContainerExecutor. The LCEResourcesHandler is a public API(exposed via yarn-site.xml) - users may have implemented their own. Removing it would mean breaking backwards compatibility.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          hi Varun Vasudev, thanks for the updated patch. Please see comments below.

          CGroupsCpuResourceHandlerImpl.java

          import org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler;
          import org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler;
          import org.apache.hadoop.yarn.server.nodemanager.util.LCEResourcesHandler;
          

          These are unused imports in CGroupsCpuResourceHandlerImpl

          @VisibleForTesting
          static final String CPU_PERIOD_US = "cfs_period_us";
          @VisibleForTesting
          static final String CPU_QUOTA_US = "cfs_quota_us";
          @VisibleForTesting
          static final String CPU_SHARES = "shares”;
          

          Move these to CGroupsHandler ?

              int quotaUS = MAX_QUOTA_US;
              int periodUS = (int) (MAX_QUOTA_US / yarnProcessors);
          

          About how shares/cfs_period_us/cfs_quota_us are used : additional comments/documentation and examples (as unit tests?) would be useful. It took me a while to trace through the code using some examples.

          CGroupsLCEResourcesHandler.java

          Since the class has been marked deprecated, is it necessary to make the rest of the changes that are included ?

          LinuxContainerExecutor.java

          private LCEResourcesHandler getResourcesHandler(Configuration conf) {
            LCEResourcesHandler handler = ReflectionUtils.newInstance(
                conf.getClass(YarnConfiguration.NM_LINUX_CONTAINER_RESOURCES_HANDLER,
                    DefaultLCEResourcesHandler.class, LCEResourcesHandler.class), conf);
          
            // Stop using CgroupsLCEResourcesHandler
            // use the resource handler chain instead
            // ResourceHandlerModule will create the cgroup cpu module if
            // CgroupsLCEResourcesHandler is set
            if (handler instanceof CgroupsLCEResourcesHandler) {
              handler =
                  ReflectionUtils.newInstance(DefaultLCEResourcesHandler.class, conf);
            }
            handler.setConf(conf);
            return handler;
          }
          

          Since all resource handling is now in the resource handler chain - there is no longer a need to have references to LCEResourcesHandler in LinuxContainerExeuctor - all related config handling etc should be in ResourceHandlerModule.java (which already seems to the case). IMO, all references to LCEResourcesHandler (and sub-classes) should be removed from LinuxContainerExecutor.

          Show
          sidharta-s Sidharta Seethana added a comment - hi Varun Vasudev , thanks for the updated patch. Please see comments below. CGroupsCpuResourceHandlerImpl.java import org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler; import org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler; import org.apache.hadoop.yarn.server.nodemanager.util.LCEResourcesHandler; These are unused imports in CGroupsCpuResourceHandlerImpl @VisibleForTesting static final String CPU_PERIOD_US = "cfs_period_us" ; @VisibleForTesting static final String CPU_QUOTA_US = "cfs_quota_us" ; @VisibleForTesting static final String CPU_SHARES = "shares”; Move these to CGroupsHandler ? int quotaUS = MAX_QUOTA_US; int periodUS = ( int ) (MAX_QUOTA_US / yarnProcessors); About how shares/cfs_period_us/cfs_quota_us are used : additional comments/documentation and examples (as unit tests?) would be useful. It took me a while to trace through the code using some examples. CGroupsLCEResourcesHandler.java Since the class has been marked deprecated, is it necessary to make the rest of the changes that are included ? LinuxContainerExecutor.java private LCEResourcesHandler getResourcesHandler(Configuration conf) { LCEResourcesHandler handler = ReflectionUtils.newInstance( conf.getClass(YarnConfiguration.NM_LINUX_CONTAINER_RESOURCES_HANDLER, DefaultLCEResourcesHandler.class, LCEResourcesHandler.class), conf); // Stop using CgroupsLCEResourcesHandler // use the resource handler chain instead // ResourceHandlerModule will create the cgroup cpu module if // CgroupsLCEResourcesHandler is set if (handler instanceof CgroupsLCEResourcesHandler) { handler = ReflectionUtils.newInstance(DefaultLCEResourcesHandler.class, conf); } handler.setConf(conf); return handler; } Since all resource handling is now in the resource handler chain - there is no longer a need to have references to LCEResourcesHandler in LinuxContainerExeuctor - all related config handling etc should be in ResourceHandlerModule.java (which already seems to the case). IMO, all references to LCEResourcesHandler (and sub-classes) should be removed from LinuxContainerExecutor.
          Hide
          vvasudev Varun Vasudev added a comment -

          Move the implementation from CgroupsLCEResourcesHandler into the new resource handler. Mark the former as deprecated.

          Fixed.

          The existing config setting remains for backward compatibility - when CgroupsLCEResourcesHandler is specified as the resource handler class, use the new resource handler implementation ( handled in ResourceHandlerModule ) . The existing integration in LinuxContainerExecutor can probably be removed at this point.

          Fixed.

          Config behavior : If the existing config setting and the new config setting are both explicitly specified and are incompatible, bail out ( this might be a bit tricky depending on how defaults are handled for these settings )

          This doesn't apply - there are no conflicting configs - the only issue was if both were specified, but with the latest patch - it ends up creating only one object.

          LOG.debug() statements should be guarded by a config check, IMO.

          Guarded the debug statement that does some string manipulation. The others just print literal strings - there should be little cost to them.

          YarnConfiguration changes : Disk/Network configs define a prefix ( e.g NM_DISK_RESOURCE_PREFIX ) - maybe something similar should be used for CPU ?

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Move the implementation from CgroupsLCEResourcesHandler into the new resource handler. Mark the former as deprecated. Fixed. The existing config setting remains for backward compatibility - when CgroupsLCEResourcesHandler is specified as the resource handler class, use the new resource handler implementation ( handled in ResourceHandlerModule ) . The existing integration in LinuxContainerExecutor can probably be removed at this point. Fixed. Config behavior : If the existing config setting and the new config setting are both explicitly specified and are incompatible, bail out ( this might be a bit tricky depending on how defaults are handled for these settings ) This doesn't apply - there are no conflicting configs - the only issue was if both were specified, but with the latest patch - it ends up creating only one object. LOG.debug() statements should be guarded by a config check, IMO. Guarded the debug statement that does some string manipulation. The others just print literal strings - there should be little cost to them. YarnConfiguration changes : Disk/Network configs define a prefix ( e.g NM_DISK_RESOURCE_PREFIX ) - maybe something similar should be used for CPU ? Fixed.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Given that one of the goals of this JIRA/patch is to deprecate the current implementation, how does this approach sound?

          • Move the implementation from CgroupsLCEResourcesHandler into the new resource handler. Mark the former as deprecated.
          • The existing config setting remains for backward compatibility - when CgroupsLCEResourcesHandler is specified as the resource handler class, use the new resource handler implementation ( handled in ResourceHandlerModule ) . The existing integration in LinuxContainerExecutor can probably be removed at this point.
          • Config behavior : If the existing config setting and the new config setting are both explicitly specified and are incompatible, bail out ( this might be a bit tricky depending on how defaults are handled for these settings )

          A couple of other minor comments based on a first look :

          • LOG.debug() statements should be guarded by a config check, IMO.
          • YarnConfiguration changes : Disk/Network configs define a prefix ( e.g NM_DISK_RESOURCE_PREFIX ) - maybe something similar should be used for CPU ?
          Show
          sidharta-s Sidharta Seethana added a comment - Given that one of the goals of this JIRA/patch is to deprecate the current implementation, how does this approach sound? Move the implementation from CgroupsLCEResourcesHandler into the new resource handler. Mark the former as deprecated. The existing config setting remains for backward compatibility - when CgroupsLCEResourcesHandler is specified as the resource handler class, use the new resource handler implementation ( handled in ResourceHandlerModule ) . The existing integration in LinuxContainerExecutor can probably be removed at this point. Config behavior : If the existing config setting and the new config setting are both explicitly specified and are incompatible, bail out ( this might be a bit tricky depending on how defaults are handled for these settings ) A couple of other minor comments based on a first look : LOG.debug() statements should be guarded by a config check, IMO. YarnConfiguration changes : Disk/Network configs define a prefix ( e.g NM_DISK_RESOURCE_PREFIX ) - maybe something similar should be used for CPU ?
          Hide
          vvasudev Varun Vasudev added a comment -

          All the existing configurations will continue to work as is. The patch adds a new configuration -

          yarn.nodemanager.resource.cpu.enabled
          

          which if set to true will create the cpu handler as part of the resource handler chain. None of the other configurations change. If both yarn.nodemanager.resource.cpu.enabled and yarn.nodemanager.linux-container-executor.resources-handler.class are set, you end up in a situation where both objects end up modifying the same file.

          Show
          vvasudev Varun Vasudev added a comment - All the existing configurations will continue to work as is. The patch adds a new configuration - yarn.nodemanager.resource.cpu.enabled which if set to true will create the cpu handler as part of the resource handler chain. None of the other configurations change. If both yarn.nodemanager.resource.cpu.enabled and yarn.nodemanager.linux-container-executor.resources-handler.class are set, you end up in a situation where both objects end up modifying the same file.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          hi Varun Vasudev,

          Thank you for the patch. I took a look at the patch and it is a bit unclear how the new configs/resource handler are meant to interact with the existing CgroupsLCEResourcesHandler . IMO, one of the goals here is to deprecate CgroupsLCEResourcesHandler and use the new resource handler mechanism so that all the resource handling/isolation is handled in a consistent manner.

          Could you please provide a description of the changes introduced in this patch and what the interaction would be with the existing CPU cgroups implementation (especially from a configuration perspective) ?

          thanks,
          -Sidharta

          Show
          sidharta-s Sidharta Seethana added a comment - hi Varun Vasudev , Thank you for the patch. I took a look at the patch and it is a bit unclear how the new configs/resource handler are meant to interact with the existing CgroupsLCEResourcesHandler . IMO, one of the goals here is to deprecate CgroupsLCEResourcesHandler and use the new resource handler mechanism so that all the resource handling/isolation is handled in a consistent manner. Could you please provide a description of the changes introduced in this patch and what the interaction would be with the existing CPU cgroups implementation (especially from a configuration perspective) ? thanks, -Sidharta
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 57s trunk passed
          +1 compile 2m 32s trunk passed with JDK v1.8.0_66
          +1 compile 2m 37s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 2m 50s trunk passed
          +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 34s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 10s the patch passed
          +1 compile 2m 52s the patch passed with JDK v1.8.0_66
          +1 javac 2m 52s the patch passed
          +1 compile 3m 5s the patch passed with JDK v1.7.0_85
          +1 javac 3m 5s the patch passed
          -1 checkstyle 0m 33s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 244, now 241).
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 2s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 25s the patch passed with JDK v1.7.0_85
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 34s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          +1 unit 9m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          63m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774788/YARN-3542.002.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9aca965b7289 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c37c3f4
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9810/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9810/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9810/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 57s trunk passed +1 compile 2m 32s trunk passed with JDK v1.8.0_66 +1 compile 2m 37s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 2m 50s trunk passed +1 javadoc 1m 17s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 34s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 10s the patch passed +1 compile 2m 52s the patch passed with JDK v1.8.0_66 +1 javac 2m 52s the patch passed +1 compile 3m 5s the patch passed with JDK v1.7.0_85 +1 javac 3m 5s the patch passed -1 checkstyle 0m 33s Patch generated 5 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 244, now 241). +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 2s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 25s the patch passed with JDK v1.7.0_85 +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 9m 34s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_66. +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. +1 unit 9m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 63m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774788/YARN-3542.002.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9aca965b7289 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37c3f4 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9810/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9810/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9810/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address test failures and checkstyle issues.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address test failures and checkstyle issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 20s trunk passed
          +1 compile 2m 32s trunk passed with JDK v1.8.0_66
          +1 compile 2m 34s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 30s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 2s the patch passed
          +1 compile 2m 32s the patch passed with JDK v1.8.0_66
          +1 javac 2m 32s the patch passed
          +1 compile 2m 35s the patch passed with JDK v1.7.0_85
          +1 javac 2m 35s the patch passed
          -1 checkstyle 0m 33s Patch generated 20 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 244, now 256).
          +1 mvnsite 1m 7s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 2s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 29s the patch passed with JDK v1.7.0_85
          -1 unit 0m 28s hadoop-yarn-api in the patch failed with JDK v1.8.0_66.
          -1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 0m 29s hadoop-yarn-api in the patch failed with JDK v1.7.0_85.
          -1 unit 10m 2s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 30s Patch does not generate ASF License warnings.
          63m 35s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler
          JDK v1.7.0_85 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774776/YARN-3542.001.patch
          JIRA Issue YARN-3542
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f7a78d50ed67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c37c3f4
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_85.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_85.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9809/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9809/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 20s trunk passed +1 compile 2m 32s trunk passed with JDK v1.8.0_66 +1 compile 2m 34s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 30s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 2s the patch passed +1 compile 2m 32s the patch passed with JDK v1.8.0_66 +1 javac 2m 32s the patch passed +1 compile 2m 35s the patch passed with JDK v1.7.0_85 +1 javac 2m 35s the patch passed -1 checkstyle 0m 33s Patch generated 20 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 244, now 256). +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 2s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 29s the patch passed with JDK v1.7.0_85 -1 unit 0m 28s hadoop-yarn-api in the patch failed with JDK v1.8.0_66. -1 unit 9m 37s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_66. -1 unit 0m 29s hadoop-yarn-api in the patch failed with JDK v1.7.0_85. -1 unit 10m 2s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 30s Patch does not generate ASF License warnings. 63m 35s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler JDK v1.7.0_85 Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774776/YARN-3542.001.patch JIRA Issue YARN-3542 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f7a78d50ed67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37c3f4 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_85.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdk1.7.0_85.txt https://builds.apache.org/job/PreCommit-YARN-Build/9809/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9809/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9809/console This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Attached an initial version of the patch.

          Show
          vvasudev Varun Vasudev added a comment - Attached an initial version of the patch.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          +1 to this

          Show
          sidharta-s Sidharta Seethana added a comment - +1 to this
          Hide
          sidharta-s Sidharta Seethana added a comment -

          +1 to this

          Show
          sidharta-s Sidharta Seethana added a comment - +1 to this
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Let's do this in the 2.8 timeline before the two implementations diverge more.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Let's do this in the 2.8 timeline before the two implementations diverge more.

            People

            • Assignee:
              vvasudev Varun Vasudev
              Reporter:
              sidharta-s Sidharta Seethana
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development