Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5797

Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add new metrics to the node manager around the local cache sizes and how much is being cleaned from them on a regular bases. For example, we can expose information contained in the LocalCacheCleanerStats class.

      1. YARN-5797-branch-2.001.patch
        27 kB
        Chris Trezzo
      2. YARN-5797-trunk.002.patch
        27 kB
        Chris Trezzo
      3. YARN-5797-trunk.003.patch
        27 kB
        Chris Trezzo
      4. YARN-5797-trunk-v1.patch
        26 kB
        Chris Trezzo

        Issue Links

          Activity

          Hide
          ctrezzo Chris Trezzo added a comment - - edited

          Attaching v1 patch to get a qa run. Summary:

          1. Added metrics to NodeManagerMetrics that will expose stats from LocalCacheCleanerStats.
          2. Adjusted ResourceLocalizationService constructor to take a NodeManagerMetrics param. Also adjusted handleCacheCleanup to update the new metrics.
          3. Adjusted TestLocalCacheCleanup to cover metrics as well.
          4. Refactored other unit tests to adjust for change in ResourceLocalizationService constructor.
          Show
          ctrezzo Chris Trezzo added a comment - - edited Attaching v1 patch to get a qa run. Summary: Added metrics to NodeManagerMetrics that will expose stats from LocalCacheCleanerStats . Adjusted ResourceLocalizationService constructor to take a NodeManagerMetrics param. Also adjusted handleCacheCleanup to update the new metrics. Adjusted TestLocalCacheCleanup to cover metrics as well. Refactored other unit tests to adjust for change in ResourceLocalizationService constructor.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 6m 58s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 40s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 5 new + 311 unchanged - 6 fixed = 316 total (was 317)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 47s the patch passed
          +1 javadoc 0m 15s the patch passed
          +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          28m 57s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5797
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836272/YARN-5797-trunk-v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d6e695eefef3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7ba74be
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13713/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13713/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13713/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 40s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 5 new + 311 unchanged - 6 fixed = 316 total (was 317) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 28m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5797 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836272/YARN-5797-trunk-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d6e695eefef3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba74be Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13713/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13713/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13713/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Note that the patch exposes the following metrics about the cache cleanup:

          1. cacheSizeBeforeClean - The local cache size (public and private) before clean in Bytes
          2. totalBytesDeleted - # of total bytes deleted from the public and private local cache
          3. publicBytesDeleted - # of bytes deleted from the public local cache
          4. privateBytesDeleted - # of bytes deleted from the private local cache

          LocalCacheCleanerStats also exposes the individual amounts deleted (in bytes) from each user private cache. I wasn't quite sure of a good way to expose this via metrics, so I left it out of the current patch.

          Show
          ctrezzo Chris Trezzo added a comment - Note that the patch exposes the following metrics about the cache cleanup: cacheSizeBeforeClean - The local cache size (public and private) before clean in Bytes totalBytesDeleted - # of total bytes deleted from the public and private local cache publicBytesDeleted - # of bytes deleted from the public local cache privateBytesDeleted - # of bytes deleted from the private local cache LocalCacheCleanerStats also exposes the individual amounts deleted (in bytes) from each user private cache. I wasn't quite sure of a good way to expose this via metrics, so I left it out of the current patch.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The patch looks good to me overall. It would be great if you can clean up the applicable checkstyle issues, and we should be good to go. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - The patch looks good to me overall. It would be great if you can clean up the applicable checkstyle issues, and we should be good to go. Thanks!
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thank you Chris Trezzo for patch. The patch looks good to me too. As Sangjin Lee mentioned have to clean up check style issues.
          Patch don't apply for branch-2 probably might have to create patch for branch-2 also.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thank you Chris Trezzo for patch. The patch looks good to me too. As Sangjin Lee mentioned have to clean up check style issues. Patch don't apply for branch-2 probably might have to create patch for branch-2 also.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Attaching v2 to fix checkstyle issues. Once I get a +1 for the trunk patch I will create another version for branch-2. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Attaching v2 to fix checkstyle issues. Once I get a +1 for the trunk patch I will create another version for branch-2. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 0m 33s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 33s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 49s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 304 unchanged - 7 fixed = 305 total (was 311)
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          +1 javadoc 0m 16s the patch passed
          +1 unit 13m 23s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          29m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5797
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843130/YARN-5797-trunk.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 516b3fe3566a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e24a923
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14289/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14289/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14289/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 8m 1s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 304 unchanged - 7 fixed = 305 total (was 311) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 13m 23s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 29m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5797 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843130/YARN-5797-trunk.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 516b3fe3566a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e24a923 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14289/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14289/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14289/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          The checkstyle warning is because the TestResourceLocalizationService#testDownloadingResourcesOnContainerKill method is over 150 lines:

          ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java:1128: @Test(timeout = 20000):3: Method length is 242 lines (max allowed is 150).

          This patch barely touches the method (i.e. changes 1 line) so I think it would be wrong to refactor this method as part of this patch. I will file a jira to address the method refactor.

          Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - The checkstyle warning is because the TestResourceLocalizationService#testDownloadingResourcesOnContainerKill method is over 150 lines: ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java:1128: @Test(timeout = 20000):3: Method length is 242 lines (max allowed is 150). This patch barely touches the method (i.e. changes 1 line) so I think it would be wrong to refactor this method as part of this patch. I will file a jira to address the method refactor. Thanks!
          Hide
          ctrezzo Chris Trezzo added a comment -

          Jira filed: YARN-6004

          Show
          ctrezzo Chris Trezzo added a comment - Jira filed: YARN-6004
          Hide
          ctrezzo Chris Trezzo added a comment -

          Bump for a review/commit. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Bump for a review/commit. Thanks!
          Hide
          ctrezzo Chris Trezzo added a comment -

          Patch is now available for YARN-6004 as well. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Patch is now available for YARN-6004 as well. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 13m 36s trunk passed
          +1 compile 0m 29s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 43s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 26s the patch passed
          +1 javac 0m 26s the patch passed
          -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 289 unchanged - 7 fixed = 290 total (was 296)
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 48s the patch passed
          +1 javadoc 0m 16s the patch passed
          +1 unit 12m 55s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          33m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5797
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843130/YARN-5797-trunk.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 94aa661c8faf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6a5516c
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15429/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15429/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15429/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 13m 36s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 289 unchanged - 7 fixed = 290 total (was 296) +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 48s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 12m 55s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 33m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5797 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843130/YARN-5797-trunk.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 94aa661c8faf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a5516c Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15429/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15429/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15429/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          As soon as the patch for YARN-6004 is in, I will rebase for trunk and branch-2.

          Show
          ctrezzo Chris Trezzo added a comment - As soon as the patch for YARN-6004 is in, I will rebase for trunk and branch-2.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Attached is a v3 patch that is rebased on trunk.

          Show
          ctrezzo Chris Trezzo added a comment - Attached is a v3 patch that is rebased on trunk.
          Hide
          ctrezzo Chris Trezzo added a comment - - edited

          Also attached is a v1 branch-2 patch. The only conflict was in TestResourceLocalizationService.java and it was due to a difference in formatting between the branches on one line.

          Show
          ctrezzo Chris Trezzo added a comment - - edited Also attached is a v1 branch-2 patch. The only conflict was in TestResourceLocalizationService.java and it was due to a difference in formatting between the branches on one line.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 13m 15s trunk passed
          +1 compile 0m 27s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 39s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          +1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 284 unchanged - 6 fixed = 284 total (was 290)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 45s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 12m 52s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          32m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5797
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862167/YARN-5797-trunk.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 10996f70376d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7d963c4
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15540/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15540/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 13m 15s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 284 unchanged - 6 fixed = 284 total (was 290) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 12m 52s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 32m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5797 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862167/YARN-5797-trunk.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 10996f70376d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7d963c4 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15540/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15540/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 6m 47s branch-2 passed
          +1 compile 0m 24s branch-2 passed with JDK v1.8.0_121
          +1 compile 0m 29s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 22s branch-2 passed
          +1 mvnsite 0m 30s branch-2 passed
          +1 mvneclipse 0m 14s branch-2 passed
          +1 findbugs 0m 53s branch-2 passed
          +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 0m 19s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.8.0_121
          +1 javac 0m 21s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.7.0_121
          +1 javac 0m 25s the patch passed
          +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 303 unchanged - 6 fixed = 303 total (was 309)
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 16s the patch passed with JDK v1.7.0_121
          -1 unit 12m 58s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          41m 41s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
          JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue YARN-5797
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862174/YARN-5797-branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11f03bcec26c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 0a7ec1f
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15542/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15542/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15542/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 6m 47s branch-2 passed +1 compile 0m 24s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 29s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 22s branch-2 passed +1 mvnsite 0m 30s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 findbugs 0m 53s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_121 +1 javadoc 0m 19s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 24s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_121 +1 javac 0m 21s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_121 +1 javac 0m 25s the patch passed +1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 303 unchanged - 6 fixed = 303 total (was 309) +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_121 -1 unit 12m 58s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 41m 41s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5797 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862174/YARN-5797-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11f03bcec26c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 0a7ec1f Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15542/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15542/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15542/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Branch-2 test failures are unrelated. From my perspective the patch is good to go for trunk and branch-2. Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Branch-2 test failures are unrelated. From my perspective the patch is good to go for trunk and branch-2. Thanks!
          Hide
          mingma Ming Ma added a comment -

          +1. Thanks Chris Trezzo for the contribution and Sangjin Lee and Bibin A Chundatt for the review. Committed to trunk and branch-2.

          Show
          mingma Ming Ma added a comment - +1. Thanks Chris Trezzo for the contribution and Sangjin Lee and Bibin A Chundatt for the review. Committed to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11544/)
          YARN-5797. Add metrics to the node manager for cleaning the PUBLIC and (mingma: rev 0116c3c95769e204ab2600510f0efd6baafb43e0)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheCleanup.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheDirectoryManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11544/ ) YARN-5797 . Add metrics to the node manager for cleaning the PUBLIC and (mingma: rev 0116c3c95769e204ab2600510f0efd6baafb43e0) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheCleanup.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheDirectoryManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
          Hide
          ctrezzo Chris Trezzo added a comment -

          Thanks everyone for the reviews and Ming Ma for the commit!

          Show
          ctrezzo Chris Trezzo added a comment - Thanks everyone for the reviews and Ming Ma for the commit!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
          YARN-5797. Add metrics to the node manager for cleaning the PUBLIC and (mingma: rev 0116c3c95769e204ab2600510f0efd6baafb43e0)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheCleanup.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheDirectoryManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) YARN-5797 . Add metrics to the node manager for cleaning the PUBLIC and (mingma: rev 0116c3c95769e204ab2600510f0efd6baafb43e0) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheCleanup.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalCacheDirectoryManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

            People

            • Assignee:
              ctrezzo Chris Trezzo
              Reporter:
              ctrezzo Chris Trezzo
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development