Details

    • Hadoop Flags:
      Reviewed

      Description

      With really large clusters, the basic DiskValidator isn't sufficient for some of the less common types of disk failures.

      Look at a new DiskValidator that could do one or more of the following:

      • Add new tests to find more problems
      • Add new metrics to at least characterize problems that we haven't predicted
      1. YARN-5529.001.patch
        16 kB
        Yufei Gu
      2. YARN-5529.002.patch
        19 kB
        Yufei Gu
      3. YARN-5529.003.patch
        19 kB
        Yufei Gu

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 9m 36s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 27s trunk passed
        +1 javadoc 0m 51s trunk passed
        +1 mvninstall 0m 39s the patch passed
        +1 compile 10m 8s the patch passed
        +1 javac 10m 8s the patch passed
        -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 12 new + 7 unchanged - 0 fixed = 19 total (was 7)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 1m 42s hadoop-common-project/hadoop-common generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        +1 javadoc 0m 53s the patch passed
        +1 unit 7m 44s hadoop-common in the patch passed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        46m 32s



        Reason Tests
        FindBugs module:hadoop-common-project/hadoop-common
          org.apache.hadoop.metrics2.lib.MutableQuantiles.getEstimator() is unsynchronized, org.apache.hadoop.metrics2.lib.MutableQuantiles.setEstimator(QuantileEstimator) is synchronized At MutableQuantiles.java:synchronized At MutableQuantiles.java:[line 149]
          Random object created and used only once in org.apache.hadoop.util.ReadWriteDiskValidator.checkStatus(File) At ReadWriteDiskValidator.java:only once in org.apache.hadoop.util.ReadWriteDiskValidator.checkStatus(File) At ReadWriteDiskValidator.java:[line 48]
          Unread field:ReadWriteDiskValidatorMetrics.java:[line 56]



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5529
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842469/YARN-5529.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fe3ce4df2859 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 13d8e55
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14239/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14239/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14239/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14239/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 9m 36s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 27s trunk passed +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 10m 8s the patch passed +1 javac 10m 8s the patch passed -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 12 new + 7 unchanged - 0 fixed = 19 total (was 7) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 42s hadoop-common-project/hadoop-common generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 javadoc 0m 53s the patch passed +1 unit 7m 44s hadoop-common in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 46m 32s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.metrics2.lib.MutableQuantiles.getEstimator() is unsynchronized, org.apache.hadoop.metrics2.lib.MutableQuantiles.setEstimator(QuantileEstimator) is synchronized At MutableQuantiles.java:synchronized At MutableQuantiles.java: [line 149]   Random object created and used only once in org.apache.hadoop.util.ReadWriteDiskValidator.checkStatus(File) At ReadWriteDiskValidator.java:only once in org.apache.hadoop.util.ReadWriteDiskValidator.checkStatus(File) At ReadWriteDiskValidator.java: [line 48]   Unread field:ReadWriteDiskValidatorMetrics.java: [line 56] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5529 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842469/YARN-5529.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fe3ce4df2859 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13d8e55 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14239/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14239/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14239/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14239/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 15m 30s trunk passed
        +1 compile 13m 0s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 9s trunk passed
        +1 mvneclipse 0m 22s trunk passed
        +1 findbugs 1m 31s trunk passed
        +1 javadoc 0m 50s trunk passed
        +1 mvninstall 0m 39s the patch passed
        +1 compile 9m 42s the patch passed
        +1 javac 9m 42s the patch passed
        -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 36s the patch passed
        +1 javadoc 0m 50s the patch passed
        -1 unit 7m 18s hadoop-common in the patch failed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        58m 5s



        Reason Tests
        Failed junit tests hadoop.util.TestBasicDiskValidator
          hadoop.http.TestSSLHttpServer
          hadoop.util.TestGenericOptionsParser
          hadoop.util.TestApplicationClassLoader



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5529
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843964/YARN-5529.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 87f9415a2030 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ef2dd7b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14368/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14368/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 30s trunk passed +1 compile 13m 0s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 9m 42s the patch passed +1 javac 9m 42s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 50s the patch passed -1 unit 7m 18s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 58m 5s Reason Tests Failed junit tests hadoop.util.TestBasicDiskValidator   hadoop.http.TestSSLHttpServer   hadoop.util.TestGenericOptionsParser   hadoop.util.TestApplicationClassLoader Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5529 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843964/YARN-5529.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 87f9415a2030 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14368/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14368/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 9s trunk passed
        +1 compile 9m 33s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 22s trunk passed
        +1 javadoc 0m 47s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 9m 15s the patch passed
        +1 javac 9m 15s the patch passed
        -0 checkstyle 0m 31s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 31s the patch passed
        +1 javadoc 0m 46s the patch passed
        -1 unit 7m 43s hadoop-common in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        50m 57s



        Reason Tests
        Failed junit tests hadoop.io.file.tfile.TestVLong
          hadoop.io.file.tfile.TestTFileByteArrays
          hadoop.util.TestJarFinder
          hadoop.fs.contract.localfs.TestLocalFSContractSetTimes
          hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
          hadoop.fs.TestFsShellCopy
          hadoop.fs.contract.localfs.TestLocalFSContractOpen
          hadoop.crypto.TestCryptoStreamsForLocalFS
          hadoop.fs.shell.TestTextCommand
          hadoop.fs.viewfs.TestFcCreateMkdirLocalFs
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal
          hadoop.fs.contract.localfs.TestLocalFSContractDelete
          hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
          hadoop.fs.TestLocalFSFileContextMainOperations
          hadoop.fs.contract.localfs.TestLocalFSContractAppend
          hadoop.fs.TestDFCachingGetSpaceUsed
          hadoop.io.file.tfile.TestTFileNoneCodecsStreams
          hadoop.fs.TestSymlinkLocalFSFileContext
          hadoop.fs.contract.rawlocal.TestRawlocalContractOpen
          hadoop.fs.TestLocalFileSystemPermission
          hadoop.fs.TestFcLocalFsPermission
          hadoop.fs.viewfs.TestChRootedFileSystem
          hadoop.fs.TestLocalFileSystem
          hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus
          hadoop.io.TestArrayFile
          hadoop.fs.TestFsShellList
          hadoop.fs.TestFileContextDeleteOnExit



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5529
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843970/YARN-5529.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e342d5eec669 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ef2dd7b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14369/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14369/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14369/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14369/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 9s trunk passed +1 compile 9m 33s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 9m 15s the patch passed +1 javac 9m 15s the patch passed -0 checkstyle 0m 31s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 31s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 43s hadoop-common in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 50m 57s Reason Tests Failed junit tests hadoop.io.file.tfile.TestVLong   hadoop.io.file.tfile.TestTFileByteArrays   hadoop.util.TestJarFinder   hadoop.fs.contract.localfs.TestLocalFSContractSetTimes   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend   hadoop.fs.TestFsShellCopy   hadoop.fs.contract.localfs.TestLocalFSContractOpen   hadoop.crypto.TestCryptoStreamsForLocalFS   hadoop.fs.shell.TestTextCommand   hadoop.fs.viewfs.TestFcCreateMkdirLocalFs   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal   hadoop.fs.contract.localfs.TestLocalFSContractDelete   hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem   hadoop.fs.TestLocalFSFileContextMainOperations   hadoop.fs.contract.localfs.TestLocalFSContractAppend   hadoop.fs.TestDFCachingGetSpaceUsed   hadoop.io.file.tfile.TestTFileNoneCodecsStreams   hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.fs.contract.rawlocal.TestRawlocalContractOpen   hadoop.fs.TestLocalFileSystemPermission   hadoop.fs.TestFcLocalFsPermission   hadoop.fs.viewfs.TestChRootedFileSystem   hadoop.fs.TestLocalFileSystem   hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus   hadoop.io.TestArrayFile   hadoop.fs.TestFsShellList   hadoop.fs.TestFileContextDeleteOnExit Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5529 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843970/YARN-5529.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e342d5eec669 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14369/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14369/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14369/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14369/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment - - edited

        A few things:

        1. Please look into the failed tests
        2. In ReadWriteDiskValidator, we create a file using Files.createTempFile and delete it near the end. If some kind of exception happens (either because we're throwing a DiskErrorException or a more generic IOException from some problem), we won't end up deleting the file. If this happens a lot, we can start filling up the disk with junk over time. We should put the delete call in a finally.
        3. Typo: throw new DiskErrorException("Data in file has bee corrupted."); It should be been.
        4. In TestReadWriteDiskValidator#testReadWriteDiskValidator, the 100 used in a few places could be made a constant, or at least a variable
        5. In TestReadWriteDiskValidator#testCheckFailures, the try-catch blocks where we're expecting a failure should fail if a DiskErrorException is not thrown, and should do some basic checking in the catch statement. e.g.
              try {
                readWriteDiskValidator.checkStatus(testDir);
                fail("some message");
              } catch (DiskErrorException e) {
                some_basic_check(e);
              }
          
        6. In TestReadWriteDiskValidator#testCheckFailures when we set the permissions to 000, might that cause a problem when trying to clean it up (i.e. mvn clean)?
        Show
        rkanter Robert Kanter added a comment - - edited A few things: Please look into the failed tests In ReadWriteDiskValidator , we create a file using Files.createTempFile and delete it near the end. If some kind of exception happens (either because we're throwing a DiskErrorException or a more generic IOException from some problem), we won't end up deleting the file. If this happens a lot, we can start filling up the disk with junk over time. We should put the delete call in a finally . Typo: throw new DiskErrorException("Data in file has bee corrupted."); It should be been . In TestReadWriteDiskValidator#testReadWriteDiskValidator , the 100 used in a few places could be made a constant, or at least a variable In TestReadWriteDiskValidator#testCheckFailures , the try-catch blocks where we're expecting a failure should fail if a DiskErrorException is not thrown, and should do some basic checking in the catch statement. e.g. try { readWriteDiskValidator.checkStatus(testDir); fail( "some message" ); } catch (DiskErrorException e) { some_basic_check(e); } In TestReadWriteDiskValidator#testCheckFailures when we set the permissions to 000 , might that cause a problem when trying to clean it up (i.e. mvn clean )?
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Robert Kanter's review. I've uploaded the new patch for all your comments.

        Show
        yufeigu Yufei Gu added a comment - Thanks Robert Kanter 's review. I've uploaded the new patch for all your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 12m 37s trunk passed
        +1 compile 9m 44s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 25s trunk passed
        +1 javadoc 0m 49s trunk passed
        +1 mvninstall 0m 41s the patch passed
        +1 compile 9m 38s the patch passed
        +1 javac 9m 38s the patch passed
        -0 checkstyle 0m 27s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
        +1 mvnsite 0m 59s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 36s the patch passed
        +1 javadoc 0m 48s the patch passed
        +1 unit 8m 2s hadoop-common in the patch passed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        51m 22s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5529
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844821/YARN-5529.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 84b2cf652c8c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c0e0ef2
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14479/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14479/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14479/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 37s trunk passed +1 compile 9m 44s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 25s trunk passed +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 9m 38s the patch passed +1 javac 9m 38s the patch passed -0 checkstyle 0m 27s hadoop-common-project/hadoop-common: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 8m 2s hadoop-common in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 51m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5529 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844821/YARN-5529.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 84b2cf652c8c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c0e0ef2 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14479/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14479/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14479/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        +1 LGTM

        Show
        rkanter Robert Kanter added a comment - +1 LGTM
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Yufei Gu. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Yufei Gu . Committed to trunk and branch-2!
        Hide
        yufeigu Yufei Gu added a comment -

        Great! Thanks Robert Kanter for the review and commit.

        Show
        yufeigu Yufei Gu added a comment - Great! Thanks Robert Kanter for the review and commit.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11064 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11064/)
        YARN-5529. Create new DiskValidator class with metrics (yufeigu via (rkanter: rev 591fb159444037bf4cb651aa1228914f5d71e1bf)

        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestReadWriteDiskValidator.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidatorMetrics.java
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskValidatorFactory.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11064 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11064/ ) YARN-5529 . Create new DiskValidator class with metrics (yufeigu via (rkanter: rev 591fb159444037bf4cb651aa1228914f5d71e1bf) (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestReadWriteDiskValidator.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidatorMetrics.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskValidatorFactory.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java

          People

          • Assignee:
            yufeigu Yufei Gu
            Reporter:
            rchiang Ray Chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development