Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9624

DataNode start slowly due to the initial DU command operations

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Make it configurable how long the cached du file is valid. Useful for rolling upgrade.

      Description

      It seems starting datanode so slowly when I am finishing migration of datanodes and restart them.I look the dn logs:

      2016-01-06 16:05:08,118 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-70097061-42f8-4c33-ac27-2a6ca21e60d4
      2016-01-06 16:05:08,118 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /home/data/data/hadoop/dfs/data/data12/current, StorageType: DISK
      2016-01-06 16:05:08,176 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
      2016-01-06 16:05:08,177 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1942012336-xx.xx.xx.xx-1406726500544
      2016-01-06 16:05:08,178 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data2/current...
      2016-01-06 16:05:08,179 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data3/current...
      2016-01-06 16:05:08,179 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data4/current...
      2016-01-06 16:05:08,179 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data5/current...
      2016-01-06 16:05:08,180 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data6/current...
      2016-01-06 16:05:08,180 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data7/current...
      2016-01-06 16:05:08,180 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data8/current...
      2016-01-06 16:05:08,180 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data9/current...
      2016-01-06 16:05:08,181 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data10/current...
      2016-01-06 16:05:08,181 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data11/current...
      2016-01-06 16:05:08,181 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on volume /home/data/data/hadoop/dfs/data/data12/current...
      2016-01-06 16:09:49,646 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data7/current: 281466ms
      2016-01-06 16:09:54,235 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data9/current: 286054ms
      2016-01-06 16:09:57,859 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data2/current: 289680ms
      2016-01-06 16:10:00,333 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data5/current: 292153ms
      2016-01-06 16:10:05,696 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data8/current: 297516ms
      2016-01-06 16:10:11,229 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data6/current: 303049ms
      2016-01-06 16:10:28,075 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data12/current: 319894ms
      2016-01-06 16:10:33,017 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data4/current: 324838ms
      2016-01-06 16:10:40,177 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data10/current: 331996ms
      2016-01-06 16:10:44,882 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data3/current: 336703ms
      2016-01-06 16:11:14,241 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1942012336-xx.xx.xx.xx-1406726500544 on /home/data/data/hadoop/dfs/data/data11/current: 366060ms
      2016-01-06 16:11:14,242 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1942012336-xx.xx.xx.xx-1406726500544: 366065ms
      

      And I know that Scanning blocks on volume and then calculating the dfsUsed costs the most of time. Because my datanode's migiration costs the much time, so that dfsUsed value can't use cache-dfsused and should be doing du operations. But actually I don't need do it again because there has no operations in these datanodes. The info is these:

      /**
         * Read in the cached DU value and return it if it is less than 600 seconds
         * old (DU update interval). Slight imprecision of dfsUsed is not critical and
         * skipping DU can significantly shorten the startup time. If the cached value
         * is not available or too old, -1 is returned.
         * */
      

      The 600 seconds is a dead code. And it looks not suitable for here.

      1. HDFS-9624.001.patch
        8 kB
        Yiqun Lin
      2. HDFS-9624.002.patch
        9 kB
        Yiqun Lin
      3. HDFS-9624.003.patch
        11 kB
        Yiqun Lin
      4. HDFS-9624.004.patch
        9 kB
        Yiqun Lin
      5. HDFS-9624.005.patch
        10 kB
        Yiqun Lin
      6. HDFS-9624.006.patch
        15 kB
        Yiqun Lin
      7. HDFS-9624.007.patch
        15 kB
        Yiqun Lin

        Activity

        Hide
        linyiqun Yiqun Lin added a comment -

        I attach a intial patch,kindly review.
        First I want use the fs.du.interval to instead of the 600 secs, but I think it's still not good when I do some operations in datanode and the time of stoping dn service will larger more than du interval time. So I define a new configuration property to instead of it in this code.

        BlockPoolSlice.java
        // Return the cached value if mtime is okay.
              if (mtime > 0 && (Time.now() - mtime < 600000L)) {
                FsDatasetImpl.LOG.info("Cached dfsUsed found for " + currentDir + ": " +
                    cachedDfsUsed);
                return cachedDfsUsed;
              }
        
        Show
        linyiqun Yiqun Lin added a comment - I attach a intial patch,kindly review. First I want use the fs.du.interval to instead of the 600 secs, but I think it's still not good when I do some operations in datanode and the time of stoping dn service will larger more than du interval time. So I define a new configuration property to instead of it in this code. BlockPoolSlice.java // Return the cached value if mtime is okay. if (mtime > 0 && (Time.now() - mtime < 600000L)) { FsDatasetImpl.LOG.info( "Cached dfsUsed found for " + currentDir + ": " + cachedDfsUsed); return cachedDfsUsed; }
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 43s trunk passed
        +1 compile 0m 46s trunk passed with JDK v1.8.0_66
        +1 compile 0m 44s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 53s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 50s trunk passed with JDK v1.7.0_91
        -1 mvninstall 0m 39s hadoop-hdfs in the patch failed.
        -1 compile 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 javac 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 compile 0m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 javac 0m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 checkstyle 0m 20s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409).
        -1 mvnsite 0m 41s hadoop-hdfs in the patch failed.
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        -1 findbugs 0m 21s hadoop-hdfs in the patch failed.
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91
        -1 unit 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 0m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        26m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780925/HDFS-9624.001.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 6c5ec3d18df4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34cd7cd
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14050/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14050/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 43s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 50s trunk passed with JDK v1.7.0_91 -1 mvninstall 0m 39s hadoop-hdfs in the patch failed. -1 compile 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 javac 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 compile 0m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 javac 0m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 checkstyle 0m 20s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409). -1 mvnsite 0m 41s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 0m 21s hadoop-hdfs in the patch failed. +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91 -1 unit 0m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 0m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 26m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780925/HDFS-9624.001.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 6c5ec3d18df4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34cd7cd Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14050/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14050/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14050/console This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Modify the testcase. And the checkstyle warning because the property's name is long, there is no need to update it.

        Show
        linyiqun Yiqun Lin added a comment - Modify the testcase. And the checkstyle warning because the property's name is long, there is no need to update it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 47s trunk passed
        +1 compile 0m 51s trunk passed with JDK v1.8.0_66
        +1 compile 0m 42s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 42s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 35s the patch passed with JDK v1.8.0_66
        +1 javac 0m 35s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_91
        +1 javac 0m 38s the patch passed
        -1 checkstyle 0m 21s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409).
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91
        -1 unit 52m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 50m 20s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        129m 13s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSShell
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.namenode.TestEditLogRace
          hadoop.hdfs.TestDFSUpgradeFromImage



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780945/HDFS-9624.002.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 1ed0fd8bd2bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6702e7d
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14052/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14052/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 47s trunk passed +1 compile 0m 51s trunk passed with JDK v1.8.0_66 +1 compile 0m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 42s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 46s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 21s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409). +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 59s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_91 -1 unit 52m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 50m 20s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 129m 13s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.TestEditLogRace   hadoop.hdfs.TestDFSUpgradeFromImage Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780945/HDFS-9624.002.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1ed0fd8bd2bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6702e7d Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14052/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14052/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14052/console This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Yeah, it would be good to make the hard-coded value configurable. The added test is nice.
        Some minors:
        1. It is tested the check occurs in the configured interval time. Add another case beyond the interval?
        2. Better to use the constant value defined in BlockPoolSlice:

        +    String DU_CACHE_FILE = "dfsUsed";
        

        3. Better to use StandardCharsets.UTF_8

        new OutputStreamWriter(new FileOutputStream(outFile), "UTF-8"))
        
        Show
        drankye Kai Zheng added a comment - Yeah, it would be good to make the hard-coded value configurable. The added test is nice. Some minors: 1. It is tested the check occurs in the configured interval time. Add another case beyond the interval? 2. Better to use the constant value defined in BlockPoolSlice : + String DU_CACHE_FILE = "dfsUsed" ; 3. Better to use StandardCharsets.UTF_8 new OutputStreamWriter( new FileOutputStream(outFile), "UTF-8" ))
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Kai Zheng for review. I update the testcase for beyond internal time and modify the minors.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Kai Zheng for review. I update the testcase for beyond internal time and modify the minors.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 55s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_66
        +1 compile 0m 44s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 51s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_66
        +1 javac 0m 38s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_91
        +1 javac 0m 41s the patch passed
        -1 checkstyle 0m 21s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 408, now 410).
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 47s the patch passed with JDK v1.7.0_91
        -1 unit 69m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 68m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        165m 20s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.TestEncryptedTransfer
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.TestClientReportBadBlock
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780993/HDFS-9624.003.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 4eee7e959bca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 52b7757
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14053/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14053/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 51s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 +1 javac 0m 38s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_91 +1 javac 0m 41s the patch passed -1 checkstyle 0m 21s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 408, now 410). +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_91 -1 unit 69m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 68m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 165m 20s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestEncryptedTransfer JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestClientReportBadBlock   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780993/HDFS-9624.003.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4eee7e959bca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52b7757 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14053/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14053/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14053/console This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Update the latest patch.

        Show
        linyiqun Yiqun Lin added a comment - Update the latest patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 30s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.8.0_66
        +1 javac 0m 42s the patch passed
        +1 compile 0m 47s the patch passed with JDK v1.7.0_91
        +1 javac 0m 47s the patch passed
        -1 checkstyle 0m 23s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409).
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
        -1 unit 56m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 52m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 25s Patch does not generate ASF License warnings.
        136m 29s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRenameWhileOpen
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.TestFSImageWithAcl



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781108/HDFS-9624.003.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 16001001faf5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 89022f8
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14063/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 30s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_91 +1 javac 0m 47s the patch passed -1 checkstyle 0m 23s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 407, now 409). +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 1m 59s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 56m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 52m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 136m 29s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFSImageWithAcl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781108/HDFS-9624.003.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 16001001faf5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89022f8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14063/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14063/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14063/console This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Yiqun for the update! Is it possible to refactor the test so the two test methods can share most of the codes?

        Show
        drankye Kai Zheng added a comment - Thanks Yiqun for the update! Is it possible to refactor the test so the two test methods can share most of the codes?
        Hide
        linyiqun Yiqun Lin added a comment -

        Kai Zheng, thanks for comments. I refactor the testcases and update the latest patch.

        Show
        linyiqun Yiqun Lin added a comment - Kai Zheng , thanks for comments. I refactor the testcases and update the latest patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 6s trunk passed
        +1 compile 0m 45s trunk passed with JDK v1.8.0_66
        +1 compile 0m 43s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 1s trunk passed
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_66
        +1 javac 0m 39s the patch passed
        +1 compile 0m 43s the patch passed with JDK v1.7.0_91
        +1 javac 0m 43s the patch passed
        -1 checkstyle 0m 22s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 408, now 410).
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 2m 8s the patch passed
        +1 javadoc 1m 12s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 52s the patch passed with JDK v1.7.0_91
        -1 unit 57m 29s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 60m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        145m 44s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          hadoop.hdfs.server.datanode.TestBlockScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781184/HDFS-9624.004.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux c6ec7399709d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ed18527
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14068/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14068/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 0m 45s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_66 +1 javac 0m 39s the patch passed +1 compile 0m 43s the patch passed with JDK v1.7.0_91 +1 javac 0m 43s the patch passed -1 checkstyle 0m 22s Patch generated 2 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 408, now 410). +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 52s the patch passed with JDK v1.7.0_91 -1 unit 57m 29s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 60m 6s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 145m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.datanode.TestBlockScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781184/HDFS-9624.004.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c6ec7399709d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed18527 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14068/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14068/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14068/console This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        hi Kai Zheng, could you have time to review my latest patch again?

        Show
        linyiqun Yiqun Lin added a comment - hi Kai Zheng , could you have time to review my latest patch again?
        Hide
        drankye Kai Zheng added a comment -

        Hi Yiqun,

        Looking at the latest patch, it looks fine with two minor issues:

        • Initial the cachedDfsUsedInternalTime larger than sleepInternalTime: Initial should be Initialize. Similar for other place.
        • The Jenkins reported checking style issues: there're lines too long.

        When you get these fixed and updated, please be patient to wait. If you're lucky, this one can be picked up soon by a HDFS committer for the review and commit. Thanks.

        Show
        drankye Kai Zheng added a comment - Hi Yiqun, Looking at the latest patch, it looks fine with two minor issues: Initial the cachedDfsUsedInternalTime larger than sleepInternalTime : Initial should be Initialize . Similar for other place. The Jenkins reported checking style issues: there're lines too long. When you get these fixed and updated, please be patient to wait. If you're lucky, this one can be picked up soon by a HDFS committer for the review and commit. Thanks.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks for comments. Update the patch, the new default property key name is long, so I don't modify them.

        Show
        linyiqun Yiqun Lin added a comment - Thanks for comments. Update the patch, the new default property key name is long, so I don't modify them.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 4s HDFS-9624 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781708/HDFS-9624.004.patch
        JIRA Issue HDFS-9624
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14097/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 4s HDFS-9624 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781708/HDFS-9624.004.patch JIRA Issue HDFS-9624 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14097/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi, thanks for working on this, patch seems good overall, a few small comments:

        • Add VisibleForTesting annotation on newly public DU_CACHE_FILE
        • Let's make some mention of rolling upgrade in the hdfs-default description, not clear to the end-user what "cache-dfsused" is or means.
        • Can we do a test that does not use a Thread.sleep? Like do some timer injection or something.
        Show
        andrew.wang Andrew Wang added a comment - Hi, thanks for working on this, patch seems good overall, a few small comments: Add VisibleForTesting annotation on newly public DU_CACHE_FILE Let's make some mention of rolling upgrade in the hdfs-default description, not clear to the end-user what "cache-dfsused" is or means. Can we do a test that does not use a Thread.sleep? Like do some timer injection or something.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Andrew Wang for comments. I found that do timer injection in this test looks not very convenient so I create block files to consume time in the test and update the patch.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Andrew Wang for comments. I found that do timer injection in this test looks not very convenient so I create block files to consume time in the test and update the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 4s HDFS-9624 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782076/HDFS-9624.005.patch
        JIRA Issue HDFS-9624
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14110/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 4s HDFS-9624 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782076/HDFS-9624.005.patch JIRA Issue HDFS-9624 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14110/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Yiqun Lin thanks for revving the patch,

        First two fixes look good, but the test needs a little more work. What I meant by timer injection is something like the org.apache.hadoop.util.Timer class, it lets you explicitly advance the time rather than having to wait for the system clock to advance. This means the test will run in milliseconds instead of seconds, which is a lot faster.

        There are some examples for how to mock Timer in other unit tests, let me know if it's still unclear though.

        Show
        andrew.wang Andrew Wang added a comment - Hi Yiqun Lin thanks for revving the patch, First two fixes look good, but the test needs a little more work. What I meant by timer injection is something like the org.apache.hadoop.util.Timer class, it lets you explicitly advance the time rather than having to wait for the system clock to advance. This means the test will run in milliseconds instead of seconds, which is a lot faster. There are some examples for how to mock Timer in other unit tests, let me know if it's still unclear though.
        Hide
        linyiqun Yiqun Lin added a comment -

        Sorry for Wang, I misunderstand your meaning of timer injection. I use the fakeTimer to the testcase and the test finished in milliseconds instead of waitting several seconds.Update the latest patch.

        Show
        linyiqun Yiqun Lin added a comment - Sorry for Wang, I misunderstand your meaning of timer injection. I use the fakeTimer to the testcase and the test finished in milliseconds instead of waitting several seconds.Update the latest patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 28s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_66
        +1 compile 0m 40s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_66
        +1 javac 0m 37s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_91
        +1 javac 0m 38s the patch passed
        -1 checkstyle 0m 22s Patch generated 4 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 543, now 546).
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 0s the patch passed
        +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
        -1 unit 51m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 50m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        127m 56s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          hadoop.hdfs.server.namenode.TestStartup
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782194/HDFS-9624.006.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux a90bbf261705 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7b0964f
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14118/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14118/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 28s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed -1 checkstyle 0m 22s Patch generated 4 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 543, now 546). +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 51m 55s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 7s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 127m 56s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782194/HDFS-9624.006.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a90bbf261705 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7b0964f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14118/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14118/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi, looks like Jenkins flagged some checkstyle issues, could you take a look? We can ignore the file length issue, but the others look fixable.

        TestStartup is failing because of HDFS-9569, so that's not related. Would appreciate if you ran the other failed tests a few times locally to confirm that they are not related to this change.

        Show
        andrew.wang Andrew Wang added a comment - Hi, looks like Jenkins flagged some checkstyle issues, could you take a look? We can ignore the file length issue, but the others look fixable. TestStartup is failing because of HDFS-9569 , so that's not related. Would appreciate if you ran the other failed tests a few times locally to confirm that they are not related to this change.
        Hide
        andrew.wang Andrew Wang added a comment -

        Also should say, otherwise looks good! +1 pending Jenkins.

        Show
        andrew.wang Andrew Wang added a comment - Also should say, otherwise looks good! +1 pending Jenkins.
        Hide
        linyiqun Yiqun Lin added a comment -

        Modify the checkstyles warnings. I test the failed test in local and these failed are all passed. And I find the test TestDFSStripedOutputStreamWithFailure will throw NPE sometimes(can see HDFS-9346). The other tests seems not related. Update the latest patch.

        Show
        linyiqun Yiqun Lin added a comment - Modify the checkstyles warnings. I test the failed test in local and these failed are all passed. And I find the test TestDFSStripedOutputStreamWithFailure will throw NPE sometimes(can see HDFS-9346 ). The other tests seems not related. Update the latest patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 55s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_66
        +1 compile 0m 42s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_66
        +1 javac 0m 37s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.7.0_91
        +1 javac 0m 37s the patch passed
        -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 543, now 543).
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
        -1 unit 51m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 49m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        128m 21s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782424/HDFS-9624.007.patch
        JIRA Issue HDFS-9624
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 1f71e79b4350 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1da762c
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14130/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14130/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 46s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_91 +1 javac 0m 37s the patch passed -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 543, now 543). +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 51m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 49m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 128m 21s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782424/HDFS-9624.007.patch JIRA Issue HDFS-9624 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1f71e79b4350 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1da762c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14130/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14130/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14130/console This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        The failed tests seem not related. The two testcases are both ok in local. And the one of failed tests TestNNThroughputBenchmark has appeared in other issues like HDFS-9639, HDFS-9648.

        Show
        linyiqun Yiqun Lin added a comment - The failed tests seem not related. The two testcases are both ok in local. And the one of failed tests TestNNThroughputBenchmark has appeared in other issues like HDFS-9639 , HDFS-9648 .
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM, will commit shortly

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM, will commit shortly
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to branch-2 and trunk, thanks for the contribution Yiqun Lin!

        Show
        andrew.wang Andrew Wang added a comment - Committed to branch-2 and trunk, thanks for the contribution Yiqun Lin !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9122 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9122/)
        HDFS-9624. DataNode start slowly due to the initial DU command (wang: rev c07f7fa8ff752436726239d938e0461236839acf)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9122 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9122/ ) HDFS-9624 . DataNode start slowly due to the initial DU command (wang: rev c07f7fa8ff752436726239d938e0461236839acf) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Andrew Wang for commit!

        Show
        linyiqun Yiqun Lin added a comment - Thanks Andrew Wang for commit!
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Should this be backport to branch-2.8? Looks like a good fix to have.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Should this be backport to branch-2.8? Looks like a good fix to have.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Yiqun Lin I updated the release note based on my understanding of this jira. Please update it if you find it imprecise. Thanks.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Yiqun Lin I updated the release note based on my understanding of this jira. Please update it if you find it imprecise. Thanks.

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            linyiqun Yiqun Lin
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development