Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3943

Use separate threshold configurations for disk-full detection and disk-not-full detection.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Use separate threshold configurations to check when disks become full and when disks become good. Currently the configuration "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage" and "yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb" are used to check both when disks become full and when disks become good. It will be better to use two configurations: one is used when disks become full from not-full and the other one is used when disks become not-full from full. So we can avoid oscillating frequently.
      For example: we can set the one for disk-full detection higher than the one for disk-not-full detection.

      1. YARN-3943.000.patch
        25 kB
        zhihai xu
      2. YARN-3943.001.patch
        20 kB
        zhihai xu
      3. YARN-3943.002.patch
        18 kB
        zhihai xu

        Activity

        Hide
        kasha Karthik Kambatla added a comment -

        Marking it critical, since some users see very poor performance when the disk utilization hovers around the max-disk-utilization mark.

        Show
        kasha Karthik Kambatla added a comment - Marking it critical, since some users see very poor performance when the disk utilization hovers around the max-disk-utilization mark.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 31s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 56s There were no new javac warning messages.
        -1 javadoc 9m 52s The applied patch generated 2 additional warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
        +1 yarn tests 7m 43s Tests passed in hadoop-yarn-server-nodemanager.
            56m 40s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12754583/YARN-3943.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6f72f1e
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/diffJavadocWarnings.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9029/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9029/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 31s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 56s There were no new javac warning messages. -1 javadoc 9m 52s The applied patch generated 2 additional warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. +1 yarn tests 7m 43s Tests passed in hadoop-yarn-server-nodemanager.     56m 40s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754583/YARN-3943.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f72f1e javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/diffJavadocWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9029/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9029/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9029/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 3s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 9m 57s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common.
        +1 yarn tests 7m 39s Tests passed in hadoop-yarn-server-nodemanager.
            56m 14s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12754660/YARN-3943.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 090d266
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9037/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9037/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 3s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common. +1 yarn tests 7m 39s Tests passed in hadoop-yarn-server-nodemanager.     56m 14s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754660/YARN-3943.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 090d266 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9037/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9037/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9037/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        I attached a new patch YARN-3943.001.patch for review. The new patch will keep backward compatibility by using the old configuration "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage" as high water mark threshold for disk-full detection and creating a new configuration "yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage" as low water mark threshold for disk-not-full detection. It also makes both configurations use same default value and if low water mark threshold is more than high water mark threshold, it will be set to the same value as high water mark threshold.

        Show
        zxu zhihai xu added a comment - I attached a new patch YARN-3943 .001.patch for review. The new patch will keep backward compatibility by using the old configuration "yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage" as high water mark threshold for disk-full detection and creating a new configuration "yarn.nodemanager.disk-health-checker.disk-utilization-watermark-low-per-disk-percentage" as low water mark threshold for disk-not-full detection. It also makes both configurations use same default value and if low water mark threshold is more than high water mark threshold, it will be set to the same value as high water mark threshold.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 51s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 8s There were no new javac warning messages.
        +1 javadoc 10m 25s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
        +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common.
        -1 yarn tests 8m 26s Tests failed in hadoop-yarn-server-nodemanager.
            59m 1s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
        Timed out tests org.apache.hadoop.yarn.server.nodemanager.containermanager.application.TestApplication



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762493/YARN-3943.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 67b0e96
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9270/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9270/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 51s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 8s There were no new javac warning messages. +1 javadoc 10m 25s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 4m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common. -1 yarn tests 8m 26s Tests failed in hadoop-yarn-server-nodemanager.     59m 1s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Timed out tests org.apache.hadoop.yarn.server.nodemanager.containermanager.application.TestApplication Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762493/YARN-3943.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 67b0e96 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9270/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9270/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9270/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 17s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 10m 39s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 56s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 3s The patch has no lines that end in whitespace.
        +1 install 1m 53s mvn install still works.
        +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse.
        +1 findbugs 5m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 19s Tests passed in hadoop-yarn-common.
        -1 yarn tests 8m 53s Tests failed in hadoop-yarn-server-nodemanager.
            60m 55s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762496/YARN-3943.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 67b0e96
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9271/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9271/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 39s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 56s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 53s mvn install still works. +1 eclipse:eclipse 0m 40s The patch built with eclipse:eclipse. +1 findbugs 5m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 19s Tests passed in hadoop-yarn-common. -1 yarn tests 8m 53s Tests failed in hadoop-yarn-server-nodemanager.     60m 55s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762496/YARN-3943.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 67b0e96 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9271/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9271/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9271/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 42s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 17s There were no new javac warning messages.
        +1 javadoc 10m 8s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
        +1 yarn tests 8m 41s Tests passed in hadoop-yarn-server-nodemanager.
            58m 39s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762506/YARN-3943.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7fe521b
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9272/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9272/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 17s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 8m 41s Tests passed in hadoop-yarn-server-nodemanager.     58m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762506/YARN-3943.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7fe521b checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9272/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9272/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9272/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 13s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 3s There were no new javac warning messages.
        +1 javadoc 10m 39s There were no new javadoc warning messages.
        -1 release audit 0m 15s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 4m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
        -1 yarn tests 2m 2s Tests failed in hadoop-yarn-common.
        -1 yarn tests 8m 45s Tests failed in hadoop-yarn-server-nodemanager.
            56m 0s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-nodemanager
        Failed unit tests hadoop.yarn.logaggregation.TestAggregatedLogsBlock
          hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765231/YARN-3943.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 29a582a
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9364/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9364/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 13s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 39s There were no new javadoc warning messages. -1 release audit 0m 15s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 4m 33s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. -1 yarn tests 2m 2s Tests failed in hadoop-yarn-common. -1 yarn tests 8m 45s Tests failed in hadoop-yarn-server-nodemanager.     56m 0s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Failed unit tests hadoop.yarn.logaggregation.TestAggregatedLogsBlock   hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765231/YARN-3943.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 29a582a Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9364/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9364/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9364/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 44s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 8m 3s There were no new javac warning messages.
        +1 javadoc 10m 27s There were no new javadoc warning messages.
        -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 4m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
        +1 yarn tests 8m 45s Tests passed in hadoop-yarn-server-nodemanager.
            58m 48s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765248/YARN-3943.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6d5713a
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9365/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9365/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 44s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 27s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 8m 45s Tests passed in hadoop-yarn-server-nodemanager.     58m 48s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765248/YARN-3943.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6d5713a Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9365/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9365/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9365/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        The checkstyle issues and release audit warnings were pre-existing.

        Show
        zxu zhihai xu added a comment - The checkstyle issues and release audit warnings were pre-existing.
        Hide
        zxu zhihai xu added a comment -

        Hi Jason Lowe, Could you help review the patch thanks?

        Show
        zxu zhihai xu added a comment - Hi Jason Lowe , Could you help review the patch thanks?
        Hide
        jlowe Jason Lowe added a comment - - edited

        Thanks for the patch, zhihai xu!

        I think this patch causes backwards compatibility problems for users who have increased the max-disk-utilization-per-disk-percentage from the default. For example, if someone changed it to 100 and only wants disks to be treated full if they actually fill up completely then this patch will silently make those disks unusable once they fill until they free up at least 10% of their space. That's not going to be good if the cluster runs the disks over 90% most of the time, as it will effectively take the disk out of rotation for a long, long time. A more compatible change would be to not set a value in yarn-default for this property. If the property is set then we use it, but if we get a null for it then we know it isn't specified and we should just use the same threshold for low as for high.

        Shouldn't we be capping the max of diskUtilizationPercentageCutoffLow based on diskUtilizationPercentageCutoffHigh rather than 100.0?

        Nit: I think using Math.max / Math.min could make the range capping more readable (certainly less redundant with the identifiers and constants), but it's not must-fix.

        Nit: "Use" should be "Using" in the following log message. "Use" implies we are giving a directive requiring the user to do it, while "Using" states the code is doing it for them.

        +      if (lowUsableSpacePercentagePerDisk > highUsableSpacePercentagePerDisk) {
        +        LOG.warn("Use " + YarnConfiguration.
        +            NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE + " as " +
        +            YarnConfiguration.NM_WM_LOW_PER_DISK_UTILIZATION_PERCENTAGE +
        +            ", because " + YarnConfiguration.
        +            NM_WM_LOW_PER_DISK_UTILIZATION_PERCENTAGE +
        +            " is not configured properly.");
        
        Show
        jlowe Jason Lowe added a comment - - edited Thanks for the patch, zhihai xu ! I think this patch causes backwards compatibility problems for users who have increased the max-disk-utilization-per-disk-percentage from the default. For example, if someone changed it to 100 and only wants disks to be treated full if they actually fill up completely then this patch will silently make those disks unusable once they fill until they free up at least 10% of their space. That's not going to be good if the cluster runs the disks over 90% most of the time, as it will effectively take the disk out of rotation for a long, long time. A more compatible change would be to not set a value in yarn-default for this property. If the property is set then we use it, but if we get a null for it then we know it isn't specified and we should just use the same threshold for low as for high. Shouldn't we be capping the max of diskUtilizationPercentageCutoffLow based on diskUtilizationPercentageCutoffHigh rather than 100.0? Nit: I think using Math.max / Math.min could make the range capping more readable (certainly less redundant with the identifiers and constants), but it's not must-fix. Nit: "Use" should be "Using" in the following log message. "Use" implies we are giving a directive requiring the user to do it, while "Using" states the code is doing it for them. + if (lowUsableSpacePercentagePerDisk > highUsableSpacePercentagePerDisk) { + LOG.warn( "Use " + YarnConfiguration. + NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE + " as " + + YarnConfiguration.NM_WM_LOW_PER_DISK_UTILIZATION_PERCENTAGE + + ", because " + YarnConfiguration. + NM_WM_LOW_PER_DISK_UTILIZATION_PERCENTAGE + + " is not configured properly." );
        Hide
        zxu zhihai xu added a comment -

        Thanks Jason Lowe! yes, the comments are great. Nice catch for the backwards compatibility problem! I uploaded a new patch YARN-3943.002.patch, which addressed all your comments, Please review it.

        Show
        zxu zhihai xu added a comment - Thanks Jason Lowe ! yes, the comments are great. Nice catch for the backwards compatibility problem! I uploaded a new patch YARN-3943 .002.patch, which addressed all your comments, Please review it.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 20m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 0s There were no new javac warning messages.
        +1 javadoc 10m 28s There were no new javadoc warning messages.
        -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
        +1 yarn tests 8m 46s Tests passed in hadoop-yarn-server-nodemanager.
            59m 2s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765505/YARN-3943.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 35affec
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9375/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9375/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 28s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 8m 46s Tests passed in hadoop-yarn-server-nodemanager.     59m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765505/YARN-3943.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 35affec Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-nodemanager.html Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9375/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9375/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9375/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        The checkstyle issues and release audit warnings for the new patch YARN-3943.002.patch were pre-existing.

        Show
        zxu zhihai xu added a comment - The checkstyle issues and release audit warnings for the new patch YARN-3943 .002.patch were pre-existing.
        Hide
        jlowe Jason Lowe added a comment -

        +1 lgtm. Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks zhihai xu! I committed this to trunk and branch-2.

        Show
        jlowe Jason Lowe added a comment - Thanks zhihai xu ! I committed this to trunk and branch-2.
        Hide
        zxu zhihai xu added a comment -

        Thanks Jason Lowe for the review and committing the patch, greatly appreciated!

        Show
        zxu zhihai xu added a comment - Thanks Jason Lowe for the review and committing the patch, greatly appreciated!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8596 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8596/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8596 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8596/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #510 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/510/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #510 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/510/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1238 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1238/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1238 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1238/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2445 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2445/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2445 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2445/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/501/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #501 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/501/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #474 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/474/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #474 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/474/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2412 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2412/)
        YARN-3943. Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2412 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2412/ ) YARN-3943 . Use separate threshold configurations for disk-full detection (jlowe: rev 8d226225d030253152494bda32708377ad0f7af7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

          People

          • Assignee:
            zxu zhihai xu
            Reporter:
            zxu zhihai xu
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development