Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9579

Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For cross DC distcp or other applications, it becomes useful to have insight as to the traffic volume for each network distance to distinguish cross-DC traffic, local-DC-remote-rack, etc.

      FileSystem's existing bytesRead metrics tracks all the bytes read. To provide additional metrics for each network distance, we can add additional metrics to FileSystem level and have DFSInputStream update the value based on the network distance between client and the datanode.

      DFSClient will resolve client machine's network location as part of its initialization. It doesn't need to resolve datanode's network location for each read as DatanodeInfo already has the info.

      There are existing HDFS specific metrics such as ReadStatistics and DFSHedgedReadMetrics. But these metrics are only accessible via DFSClient or DFSInputStream. Not something that application framework such as MR and Tez can get to. That is the benefit of storing these new metrics in FileSystem.Statistics.

      This jira only includes metrics generation by HDFS. The consumption of these metrics at MR and Tez will be tracked by separated jiras.

      We can add similar metrics for HDFS write scenario later if it is necessary.

      1. HDFS-9579.patch
        15 kB
        Ming Ma
      2. HDFS-9579-10.patch
        42 kB
        Ming Ma
      3. HDFS-9579-2.patch
        16 kB
        Ming Ma
      4. HDFS-9579-3.patch
        17 kB
        Ming Ma
      5. HDFS-9579-4.patch
        17 kB
        Ming Ma
      6. HDFS-9579-5.patch
        19 kB
        Ming Ma
      7. HDFS-9579-6.patch
        19 kB
        Ming Ma
      8. HDFS-9579-7.patch
        39 kB
        Ming Ma
      9. HDFS-9579-8.patch
        41 kB
        Ming Ma
      10. HDFS-9579-9.patch
        42 kB
        Ming Ma
      11. HDFS-9579-branch-2.patch
        40 kB
        Ming Ma
      12. MR job counters.png
        133 kB
        Ming Ma

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9773 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9773/)
          HDFS-10208. Addendum for HDFS-9579: to handle the case when client (sjlee: rev 61f46be071e42f9eb49a54b1bd2e54feac59f808)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9773 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9773/ ) HDFS-10208 . Addendum for HDFS-9579 : to handle the case when client (sjlee: rev 61f46be071e42f9eb49a54b1bd2e54feac59f808) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          ok..you can handle as part of HDFS-10208.

          Show
          brahmareddy Brahma Reddy Battula added a comment - ok..you can handle as part of HDFS-10208 .
          Hide
          mingma Ming Ma added a comment -

          Brahma Reddy Battula, that is right for the case where ScriptBasedMapping is used but the topology script isn't set. Should we handle the special case where DEFAULT_RACK is returned from DNSToSwitchMapping#resolve similar to HDFS-10208?

          Show
          mingma Ming Ma added a comment - Brahma Reddy Battula , that is right for the case where ScriptBasedMapping is used but the topology script isn't set. Should we handle the special case where DEFAULT_RACK is returned from DNSToSwitchMapping#resolve similar to HDFS-10208 ?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          After this in, I can see , there is one extra log for each client operation " Adding a new node: "

          BLR1000006554:/home/Trunk/hadoop/bin # ./hdfs dfs -put hadoop /test2
          16/03/26 15:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          16/03/26 15:07:23 INFO net.NetworkTopology: Adding a new node: /default-rack/BLR1000006554 

          If the ScriptBasedMapping is used, then topology script should be configured and placed in all machines wherever HDFS clients created to get the correct values.It will still work, but will not have correct statistics.Since everytime client is treated will be resolved as DEFAULT_RACK

          Show
          brahmareddy Brahma Reddy Battula added a comment - After this in, I can see , there is one extra log for each client operation " Adding a new node: " BLR1000006554:/home/Trunk/hadoop/bin # ./hdfs dfs -put hadoop /test2 16/03/26 15:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/03/26 15:07:23 INFO net.NetworkTopology: Adding a new node: /default-rack/BLR1000006554 If the ScriptBasedMapping is used, then topology script should be configured and placed in all machines wherever HDFS clients created to get the correct values.It will still work, but will not have correct statistics.Since everytime client is treated will be resolved as DEFAULT_RACK
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed the branch-2 patch onto branch-2. I am not committing it to branch-2.8 because I understand 2.8 needs to be stabilized and I don't think it is critical this makes 2.8.0. Do let me know if you feel strongly that this should be in 2.8.0.

          Show
          sjlee0 Sangjin Lee added a comment - Committed the branch-2 patch onto branch-2. I am not committing it to branch-2.8 because I understand 2.8 needs to be stabilized and I don't think it is critical this makes 2.8.0. Do let me know if you feel strongly that this should be in 2.8.0.
          Hide
          mingma Ming Ma added a comment -

          The branch-2 version also applies to branch-2.8. Unless others ask for it, skip the work for branch-2.7 and branch-2.6 given this is a feature enhancement and requires extra effort.

          Show
          mingma Ming Ma added a comment - The branch-2 version also applies to branch-2.8. Unless others ask for it, skip the work for branch-2.7 and branch-2.6 given this is a feature enhancement and requires extra effort.
          Hide
          mingma Ming Ma added a comment -

          Thank you Sangjin Lee, Mingliang Liu and Colin P. McCabe! Here is the patch for branch-2 which passed all tests under hadoop-common-project and hadoop-hdfs-project locally.

          Show
          mingma Ming Ma added a comment - Thank you Sangjin Lee , Mingliang Liu and Colin P. McCabe ! Here is the patch for branch-2 which passed all tests under hadoop-common-project and hadoop-hdfs-project locally.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9478 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9478/)
          HDFS-9579. Provide bytes-read-by-network-distance metrics at (sjlee: rev cd8b6889a74a949e37f4b2eb664cdf3b59bfb93b)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReader.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9478 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9478/ ) HDFS-9579 . Provide bytes-read-by-network-distance metrics at (sjlee: rev cd8b6889a74a949e37f4b2eb664cdf3b59bfb93b) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReader.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed the patch to trunk. It appears it does not apply cleanly to branch-2 and down. Ming Ma, could you please provide a patch for each of the branches you'd like to propagate this to? Thanks Ming for your contribution, and Mingliang Liu for your review!

          Show
          sjlee0 Sangjin Lee added a comment - Committed the patch to trunk. It appears it does not apply cleanly to branch-2 and down. Ming Ma , could you please provide a patch for each of the branches you'd like to propagate this to? Thanks Ming for your contribution, and Mingliang Liu for your review!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 37s trunk passed
          +1 compile 5m 47s trunk passed with JDK v1.8.0_74
          +1 compile 6m 39s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 2m 19s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 5m 10s trunk passed
          +1 javadoc 2m 17s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 12s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 2s the patch passed
          +1 compile 5m 41s the patch passed with JDK v1.8.0_74
          +1 javac 5m 41s the patch passed
          +1 compile 6m 37s the patch passed with JDK v1.7.0_95
          +1 javac 6m 37s the patch passed
          -1 checkstyle 1m 11s root: patch generated 4 new + 546 unchanged - 10 fixed = 550 total (was 556)
          +1 mvnsite 2m 19s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 48s the patch passed
          +1 javadoc 2m 16s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 14s the patch passed with JDK v1.7.0_95
          -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.8.0_74.
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 55m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 6m 58s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 53m 49s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          -1 asflicense 0m 25s Patch generated 2 ASF License warnings.
          190m 48s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794287/HDFS-9579-10.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bb7dea746a0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 33239c9
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14872/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14872/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 37s trunk passed +1 compile 5m 47s trunk passed with JDK v1.8.0_74 +1 compile 6m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 19s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 5m 10s trunk passed +1 javadoc 2m 17s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 12s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 5m 41s the patch passed with JDK v1.8.0_74 +1 javac 5m 41s the patch passed +1 compile 6m 37s the patch passed with JDK v1.7.0_95 +1 javac 6m 37s the patch passed -1 checkstyle 1m 11s root: patch generated 4 new + 546 unchanged - 10 fixed = 550 total (was 556) +1 mvnsite 2m 19s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 48s the patch passed +1 javadoc 2m 16s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_95 -1 unit 6m 32s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 55m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 6m 58s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 53m 49s hadoop-hdfs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 25s Patch generated 2 ASF License warnings. 190m 48s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.util.TestNativeLibraryChecker Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794287/HDFS-9579-10.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb7dea746a0e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 33239c9 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14872/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/14872/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14872/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the updated patch. I'm +1 pending jenkins. I'll commit it once jenkins comes back clean.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the updated patch. I'm +1 pending jenkins. I'll commit it once jenkins comes back clean.
          Hide
          mingma Ming Ma added a comment -

          Sangjin Lee Good point! Sangjin and I had offline discussion and it turns out ReplicaAccessor is an abstract class (Colin P. McCabe mentioned the motivation in HDFS-8924) and we can provide a default implementation for the new method added. In that way, we can keep the compatibility and allow other implementations of ReplicaAccessor to override it. Here is the new patch.

          Show
          mingma Ming Ma added a comment - Sangjin Lee Good point! Sangjin and I had offline discussion and it turns out ReplicaAccessor is an abstract class ( Colin P. McCabe mentioned the motivation in HDFS-8924 ) and we can provide a default implementation for the new method added. In that way, we can keep the compatibility and allow other implementations of ReplicaAccessor to override it. Here is the new patch.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The latest patch looks good, but ReplicaAccessor adds a new abstract method. Since it's declared as public/stable, this does raise the question of compatibility. I'm not too familiar with that class and it's not clear whether it was meant to be implemented only by Hadoop code or can be implemented by users/libraries. Colin P. McCabe, thoughts on this?

          Show
          sjlee0 Sangjin Lee added a comment - The latest patch looks good, but ReplicaAccessor adds a new abstract method. Since it's declared as public/stable, this does raise the question of compatibility. I'm not too familiar with that class and it's not clear whether it was meant to be implemented only by Hadoop code or can be implemented by users/libraries. Colin P. McCabe , thoughts on this?
          Hide
          mingma Ming Ma added a comment -

          Thanks Mingliang Liu! Any comments from others?

          Show
          mingma Ming Ma added a comment - Thanks Mingliang Liu ! Any comments from others?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 6m 54s trunk passed
          +1 compile 5m 50s trunk passed with JDK v1.8.0_74
          +1 compile 6m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 2m 23s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 4s trunk passed
          +1 javadoc 2m 18s trunk passed with JDK v1.8.0_74
          +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 56s the patch passed
          +1 compile 5m 35s the patch passed with JDK v1.8.0_74
          +1 javac 5m 35s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_95
          +1 javac 6m 32s the patch passed
          -1 checkstyle 1m 11s root: patch generated 4 new + 546 unchanged - 10 fixed = 550 total (was 556)
          +1 mvnsite 2m 19s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 48s the patch passed
          +1 javadoc 2m 17s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95
          +1 unit 6m 51s hadoop-common in the patch passed with JDK v1.8.0_74.
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74.
          -1 unit 53m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 51m 44s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          201m 39s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791279/HDFS-9579-9.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fac1ff19abc6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e08861
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14721/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14721/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 5m 50s trunk passed with JDK v1.8.0_74 +1 compile 6m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 2m 23s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 4s trunk passed +1 javadoc 2m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 3m 13s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 5m 35s the patch passed with JDK v1.8.0_74 +1 javac 5m 35s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_95 +1 javac 6m 32s the patch passed -1 checkstyle 1m 11s root: patch generated 4 new + 546 unchanged - 10 fixed = 550 total (was 556) +1 mvnsite 2m 19s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 48s the patch passed +1 javadoc 2m 17s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95 +1 unit 6m 51s hadoop-common in the patch passed with JDK v1.8.0_74. +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_74. -1 unit 53m 31s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 51m 44s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 201m 39s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791279/HDFS-9579-9.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fac1ff19abc6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e08861 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14721/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14721/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14721/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1 (non-binding) pending on Jenkins.

          Show
          liuml07 Mingliang Liu added a comment - +1 (non-binding) pending on Jenkins.
          Hide
          mingma Ming Ma added a comment -

          Thank you Mingliang Liu for the good catch! Here is the new patch with your suggestions.

          Show
          mingma Ming Ma added a comment - Thank you Mingliang Liu for the good catch! Here is the new patch with your suggestions.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for updating the patch, Ming Ma! I like the idea to make BlockReader distance aware and to make ReaderStrategy#doRead update the stats in base class.

          1. Should we remove the stats updating from DFSStripedInputStream as ReaderStrategy#doRead is implicitly called?
            DFSStripedInputStream.java#readWithStrategy()
                    if (dfsClient.stats != null) {
                      dfsClient.stats.incrementBytesRead(result);
                    }
            
          2. Nit: After 1 is fixed, perhaps we can make DFSClient.stat private. I'm not sure about this but: a) It's FileSystem#Statistics that is accessed by up level frameworks b) It's not used elsewhere
          3. Nit: RemoteBlockReader#networkDistance and RemoteBlockReader2#networkDistance seem final.
          Show
          liuml07 Mingliang Liu added a comment - Thanks for updating the patch, Ming Ma ! I like the idea to make BlockReader distance aware and to make ReaderStrategy#doRead update the stats in base class. Should we remove the stats updating from DFSStripedInputStream as ReaderStrategy#doRead is implicitly called? DFSStripedInputStream.java#readWithStrategy() if (dfsClient.stats != null ) { dfsClient.stats.incrementBytesRead(result); } Nit: After 1 is fixed, perhaps we can make DFSClient.stat private. I'm not sure about this but: a) It's FileSystem#Statistics that is accessed by up level frameworks b) It's not used elsewhere Nit: RemoteBlockReader#networkDistance and RemoteBlockReader2#networkDistance seem final .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 35s trunk passed
          +1 compile 5m 44s trunk passed with JDK v1.8.0_72
          +1 compile 6m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 11s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 12s trunk passed
          +1 javadoc 2m 17s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 59s the patch passed
          +1 compile 5m 46s the patch passed with JDK v1.8.0_72
          +1 javac 5m 46s the patch passed
          +1 compile 6m 45s the patch passed with JDK v1.7.0_95
          +1 javac 6m 45s the patch passed
          -1 checkstyle 1m 9s root: patch generated 4 new + 518 unchanged - 9 fixed = 522 total (was 527)
          +1 mvnsite 2m 27s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 55s the patch passed
          +1 javadoc 2m 20s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 13s the patch passed with JDK v1.7.0_95
          +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 53m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 7m 8s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 unit 53m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          190m 1s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.TestDFSUpgradeFromImage
          JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791048/HDFS-9579-8.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 269a02d9194b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14697/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14697/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 35s trunk passed +1 compile 5m 44s trunk passed with JDK v1.8.0_72 +1 compile 6m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 11s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 12s trunk passed +1 javadoc 2m 17s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 59s the patch passed +1 compile 5m 46s the patch passed with JDK v1.8.0_72 +1 javac 5m 46s the patch passed +1 compile 6m 45s the patch passed with JDK v1.7.0_95 +1 javac 6m 45s the patch passed -1 checkstyle 1m 9s root: patch generated 4 new + 518 unchanged - 9 fixed = 522 total (was 527) +1 mvnsite 2m 27s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 55s the patch passed +1 javadoc 2m 20s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 13s the patch passed with JDK v1.7.0_95 +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 53m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 7m 8s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 53m 9s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 190m 1s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.TestDFSUpgradeFromImage JDK v1.7.0_95 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791048/HDFS-9579-8.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 269a02d9194b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14697/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14697/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14697/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          New patch to address some of checkstyle and unit test issues.

          Show
          mingma Ming Ma added a comment - New patch to address some of checkstyle and unit test issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 5m 55s trunk passed with JDK v1.8.0_72
          +1 compile 6m 39s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 2m 29s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 8s trunk passed
          +1 javadoc 2m 21s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 12s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 56s the patch passed
          +1 compile 5m 49s the patch passed with JDK v1.8.0_72
          -1 javac 7m 59s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739)
          +1 javac 5m 49s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_95
          -1 javac 14m 38s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 734 unchanged - 0 fixed = 735 total (was 734)
          +1 javac 6m 39s the patch passed
          -1 checkstyle 1m 9s root: patch generated 38 new + 525 unchanged - 4 fixed = 563 total (was 529)
          +1 mvnsite 2m 24s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 47s the patch passed
          +1 javadoc 2m 21s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 14s the patch passed with JDK v1.7.0_95
          -1 unit 19m 28s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 57m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 51m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          204m 33s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestBlockReaderLocal
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
            org.apache.hadoop.hdfs.web.TestWebHDFSForHA
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestBlockReaderLocal



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790969/HDFS-9579-7.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2f9ed3552c87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5abf051
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt
          javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14688/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14688/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 5m 55s trunk passed with JDK v1.8.0_72 +1 compile 6m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 29s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 8s trunk passed +1 javadoc 2m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 12s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 5m 49s the patch passed with JDK v1.8.0_72 -1 javac 7m 59s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 739 unchanged - 0 fixed = 740 total (was 739) +1 javac 5m 49s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 -1 javac 14m 38s root-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 734 unchanged - 0 fixed = 735 total (was 734) +1 javac 6m 39s the patch passed -1 checkstyle 1m 9s root: patch generated 38 new + 525 unchanged - 4 fixed = 563 total (was 529) +1 mvnsite 2m 24s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 47s the patch passed +1 javadoc 2m 21s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_95 -1 unit 19m 28s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 57m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 51m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 204m 33s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestBlockReaderLocal JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle   org.apache.hadoop.hdfs.web.TestWebHDFSForHA JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestBlockReaderLocal Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790969/HDFS-9579-7.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2f9ed3552c87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5abf051 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_72.txt javac root-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_95.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14688/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14688/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14688/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks Mingliang Liu for the good suggestions. Here is the patch that has addressed these issues.

          To support erasure coding, the new patch moves the stats update to ReaderStrategy#doRead so that it can be used for both scenarios. This change requires updates to BlockReader and ReplicaAccessor. This might be a better fit anyway as it extends the old isLocal to the new getNetworkDistance.

          Show
          mingma Ming Ma added a comment - Thanks Mingliang Liu for the good suggestions. Here is the patch that has addressed these issues. To support erasure coding, the new patch moves the stats update to ReaderStrategy#doRead so that it can be used for both scenarios. This change requires updates to BlockReader and ReplicaAccessor . This might be a better fit anyway as it extends the old isLocal to the new getNetworkDistance .
          Hide
          liuml07 Mingliang Liu added a comment -

          The latest patch looks pretty good to me overall. Nice work. Thanks for reporting and working on this, Ming Ma.

          +1 (non-binding) after the following comments are addressed/considered.

          1. DFSClient#stats is package local mainly because it's used in DFSInputStream for updating the read bytes counter. I'm wondering if we can delegate the updateFileSystemReadStats logic to DFSClient. This way the stats is confined in DFSClient and thus transparent to DFSInputStream.
          2. As the erasure coding merged in trunk branch, we have to maintain the DFSStripeInputStream along with DFSInputStream. In this case, we need update DFSStripeInputStream#readWithStrategy as well, I suppose?
          3. Nit for reducing failing message confusion:

          - assertEquals(cluster.getDistance(node1, node2), 0);
          + assertEquals(0, cluster.getDistance(node1, node2));
          

          See http://junit.org/javadoc/latest/org/junit/Assert.html#assertEquals(java.lang.Object, java.lang.Object)

          Show
          liuml07 Mingliang Liu added a comment - The latest patch looks pretty good to me overall. Nice work. Thanks for reporting and working on this, Ming Ma . +1 (non-binding) after the following comments are addressed/considered. 1. DFSClient#stats is package local mainly because it's used in DFSInputStream for updating the read bytes counter. I'm wondering if we can delegate the updateFileSystemReadStats logic to DFSClient . This way the stats is confined in DFSClient and thus transparent to DFSInputStream . 2. As the erasure coding merged in trunk branch, we have to maintain the DFSStripeInputStream along with DFSInputStream . In this case, we need update DFSStripeInputStream#readWithStrategy as well, I suppose? 3. Nit for reducing failing message confusion: - assertEquals(cluster.getDistance(node1, node2), 0); + assertEquals(0, cluster.getDistance(node1, node2)); See http://junit.org/javadoc/latest/org/junit/Assert.html#assertEquals(java.lang.Object, java.lang.Object)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 45s Maven dependency ordering for branch
          +1 mvninstall 7m 35s trunk passed
          +1 compile 9m 0s trunk passed with JDK v1.8.0_72
          +1 compile 8m 16s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 11s trunk passed
          +1 mvnsite 2m 47s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 5m 27s trunk passed
          +1 javadoc 2m 48s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 41s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 15s the patch passed
          +1 compile 8m 28s the patch passed with JDK v1.8.0_72
          +1 javac 8m 28s the patch passed
          +1 compile 7m 48s the patch passed with JDK v1.7.0_95
          +1 javac 7m 48s the patch passed
          -1 checkstyle 1m 11s root: patch generated 32 new + 402 unchanged - 0 fixed = 434 total (was 402)
          +1 mvnsite 2m 45s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 6m 3s the patch passed
          +1 javadoc 2m 37s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 34s the patch passed with JDK v1.7.0_95
          +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 80m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 9m 27s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 102m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 41s Patch does not generate ASF License warnings.
          285m 0s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786994/HDFS-9579-6.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 102e57b54373 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acac729
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14433/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14433/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 7m 35s trunk passed +1 compile 9m 0s trunk passed with JDK v1.8.0_72 +1 compile 8m 16s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 11s trunk passed +1 mvnsite 2m 47s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 5m 27s trunk passed +1 javadoc 2m 48s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 41s trunk passed with JDK v1.7.0_95 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 15s the patch passed +1 compile 8m 28s the patch passed with JDK v1.8.0_72 +1 javac 8m 28s the patch passed +1 compile 7m 48s the patch passed with JDK v1.7.0_95 +1 javac 7m 48s the patch passed -1 checkstyle 1m 11s root: patch generated 32 new + 402 unchanged - 0 fixed = 434 total (was 402) +1 mvnsite 2m 45s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 6m 3s the patch passed +1 javadoc 2m 37s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 34s the patch passed with JDK v1.7.0_95 +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 80m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 9m 27s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 102m 14s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 41s Patch does not generate ASF License warnings. 285m 0s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786994/HDFS-9579-6.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 102e57b54373 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acac729 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14433/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14433/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks Sangjin Lee again for the review. Thanks Mingliang Liu for the input about the test results. Here is the new patch based on the suggestions. Answers inline for some of the suggestions.

          why does this need to be public now?

          In the MR change of the bigger patch, I refactored MR code that rely on this method being public. Given that patch hasn't been published yet, I have removed it in the new patch. We can update that later in the MR patch if necessary.

          if we're adopting a test using equals(), shouldn't the following code later in the same method be fixed too?

          Good point. The reason is specific to NetworkTopology implementation. When a new leaf node is added to a NetworkTopology, inner node will be reused if it already exists in the tree; while different leaf node objects with the same path can exist in the same tree. getWeight function has the same issue. Not sure if we really need to fix it.

          why is a call to update the stats added here (actualGetFromOneDataNode()) instead of the previous location (pread())?

          In order to update metrics by distance, it has to be done inside actualGetFromOneDataNode in order to access the datanode info. Given it is better to put the original incrementBytesRead and the new distance-based metrics together, the patch has actualGetFromOneDataNode call the new updateFileSystemReadStats which process both old and new read metrics.

          I suppose we need to add trivial overrides for equals() and hashCode() to address the findbugs issue...

          That reminds me why the functions were there in earlier patches.

          Show
          mingma Ming Ma added a comment - Thanks Sangjin Lee again for the review. Thanks Mingliang Liu for the input about the test results. Here is the new patch based on the suggestions. Answers inline for some of the suggestions. why does this need to be public now? In the MR change of the bigger patch, I refactored MR code that rely on this method being public. Given that patch hasn't been published yet, I have removed it in the new patch. We can update that later in the MR patch if necessary. if we're adopting a test using equals(), shouldn't the following code later in the same method be fixed too? Good point. The reason is specific to NetworkTopology implementation. When a new leaf node is added to a NetworkTopology, inner node will be reused if it already exists in the tree; while different leaf node objects with the same path can exist in the same tree. getWeight function has the same issue. Not sure if we really need to fix it. why is a call to update the stats added here (actualGetFromOneDataNode()) instead of the previous location (pread())? In order to update metrics by distance, it has to be done inside actualGetFromOneDataNode in order to access the datanode info. Given it is better to put the original incrementBytesRead and the new distance-based metrics together, the patch has actualGetFromOneDataNode call the new updateFileSystemReadStats which process both old and new read metrics. I suppose we need to add trivial overrides for equals() and hashCode() to address the findbugs issue... That reminds me why the functions were there in earlier patches.
          Hide
          liuml07 Mingliang Liu added a comment -

          The test results seem good to me. Specifically, hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs should succeed with patch HDFS-9730, which is committed after the Jenkins run. I tested locally, hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs is able to pass in trunk w. and w/o. this patch.

          Show
          liuml07 Mingliang Liu added a comment - The test results seem good to me. Specifically, hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs should succeed with patch HDFS-9730 , which is committed after the Jenkins run. I tested locally, hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs is able to pass in trunk w. and w/o. this patch.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for updating the patch Ming Ma!

          Could you confirm that the unit test failures are unrelated to the patch?

          (FileSystem.java)

          • l.3026: why does this need to be public now? I don't see a need for this to become public?
          • l.3297: a slightly more optimal version would be to use a switch-case
          • l.3434: same as above

          (NetworkTopology.java)

          • if we're adopting a test using equals(), shouldn't the following code later in the same method be fixed too?
                  while(n1!=null && n2!=null && n1.getParent()!=n2.getParent()) {
                    n1=n1.getParent();
                    n2=n2.getParent();
                    dis+=2;
                  }
            
          • l.1226: why is a call to update the stats added here (actualGetFromOneDataNode()) instead of the previous location (pread())?
          • I suppose we need to add trivial overrides for equals() and hashCode() to address the findbugs issue...
          Show
          sjlee0 Sangjin Lee added a comment - Thanks for updating the patch Ming Ma ! Could you confirm that the unit test failures are unrelated to the patch? (FileSystem.java) l.3026: why does this need to be public now? I don't see a need for this to become public? l.3297: a slightly more optimal version would be to use a switch-case l.3434: same as above (NetworkTopology.java) if we're adopting a test using equals() , shouldn't the following code later in the same method be fixed too? while (n1!= null && n2!= null && n1.getParent()!=n2.getParent()) { n1=n1.getParent(); n2=n2.getParent(); dis+=2; } l.1226: why is a call to update the stats added here ( actualGetFromOneDataNode() ) instead of the previous location ( pread() )? I suppose we need to add trivial overrides for equals() and hashCode() to address the findbugs issue...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 24s Maven dependency ordering for branch
          +1 mvninstall 6m 22s trunk passed
          +1 compile 5m 51s trunk passed with JDK v1.8.0_66
          +1 compile 6m 42s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 8s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 3s trunk passed
          +1 javadoc 2m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 32s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 26s Maven dependency ordering for patch
          +1 mvninstall 2m 13s the patch passed
          +1 compile 7m 9s the patch passed with JDK v1.8.0_66
          +1 javac 7m 9s the patch passed
          +1 compile 7m 13s the patch passed with JDK v1.7.0_91
          +1 javac 7m 13s the patch passed
          -1 checkstyle 1m 10s root: patch generated 4 new + 403 unchanged - 0 fixed = 407 total (was 403)
          +1 mvnsite 2m 39s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 59s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 2m 30s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 36s the patch passed with JDK v1.7.0_91
          +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 56m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 6m 48s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 52m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          196m 29s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.net.NetworkTopology$InnerNode doesn't override NodeBase.equals(Object) At NetworkTopology.java:At NetworkTopology.java:[line 1]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.TestReconstructStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786300/HDFS-9579-5.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cec0e0168cd5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 22a2b22
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14386/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14386/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 24s Maven dependency ordering for branch +1 mvninstall 6m 22s trunk passed +1 compile 5m 51s trunk passed with JDK v1.8.0_66 +1 compile 6m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 8s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 3s trunk passed +1 javadoc 2m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 32s trunk passed with JDK v1.7.0_91 0 mvndep 0m 26s Maven dependency ordering for patch +1 mvninstall 2m 13s the patch passed +1 compile 7m 9s the patch passed with JDK v1.8.0_66 +1 javac 7m 9s the patch passed +1 compile 7m 13s the patch passed with JDK v1.7.0_91 +1 javac 7m 13s the patch passed -1 checkstyle 1m 10s root: patch generated 4 new + 403 unchanged - 0 fixed = 407 total (was 403) +1 mvnsite 2m 39s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 59s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 36s the patch passed with JDK v1.7.0_91 +1 unit 7m 7s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 56m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 6m 48s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 52m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 196m 29s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.net.NetworkTopology$InnerNode doesn't override NodeBase.equals(Object) At NetworkTopology.java:At NetworkTopology.java: [line 1] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestReconstructStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786300/HDFS-9579-5.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cec0e0168cd5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22a2b22 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14386/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14386/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Here is the updated patch based on the above discussion.

          Show
          mingma Ming Ma added a comment - Here is the updated patch based on the above discussion.
          Hide
          sjlee0 Sangjin Lee added a comment -

          That sounds reasonable to me. I hope the names don't get any longer than that.

          Show
          sjlee0 Sangjin Lee added a comment - That sounds reasonable to me. I hope the names don't get any longer than that.
          Hide
          mingma Ming Ma added a comment -

          Thanks Colin P. McCabe. Yes, it is possible to have distance of three or five under temporary failure scenarios in certain network topology. I don't know if we really need to support it at this point given NetworkTopology and topology script are static (there is no change after it is built). Another option is to use something like bytesReadDistanceOfOneOrTwo. Thoughts?

          Show
          mingma Ming Ma added a comment - Thanks Colin P. McCabe . Yes, it is possible to have distance of three or five under temporary failure scenarios in certain network topology. I don't know if we really need to support it at this point given NetworkTopology and topology script are static (there is no change after it is built). Another option is to use something like bytesReadDistanceOfOneOrTwo . Thoughts?
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for this patch, Ming Ma. As Sangjin Lee commented, we need to use something thread-safe, so HashMap is not the right choice.

          I think it's reasonable to use individual long values as you've proposed. if it only goes up to six, should it be "six or longer"?

          This might be a dumb question, but is it possible to have a distance of one or three?

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for this patch, Ming Ma . As Sangjin Lee commented, we need to use something thread-safe, so HashMap is not the right choice. I think it's reasonable to use individual long values as you've proposed. if it only goes up to six, should it be "six or longer"? This might be a dumb question, but is it possible to have a distance of one or three?
          Hide
          mingma Ming Ma added a comment -

          Thanks Sangjin Lee! Good point about the thread visibility issue. The reason I ended up using the map is to make the code more general to support any network distance value without code change. However due to the fact that the available network distance values don't change often, using individual long variables seems ok given it addresses the issues you mentioned above.

          To use individual long variables, it could be something like below. Note that it assume tree-based topology; and it should cover the common scenarios. If we need to track network distance values, we can update it later. In addition, this means bytesReadDistanceOfFour and bytesReadDistanceOfSix won't be used for small network topology.

          volatile long bytesReadLocalHost;
          volatile long bytesReadDistanceOfTwo; // local rack case.
          volatile long bytesReadDistanceOfFour; // first-degree remote rack
          volatile long bytesReadDistanceOfSix; // second-degree remote rack
          

          I will update the patch once we agree on the new approach.

          Show
          mingma Ming Ma added a comment - Thanks Sangjin Lee ! Good point about the thread visibility issue. The reason I ended up using the map is to make the code more general to support any network distance value without code change. However due to the fact that the available network distance values don't change often, using individual long variables seems ok given it addresses the issues you mentioned above. To use individual long variables, it could be something like below. Note that it assume tree-based topology; and it should cover the common scenarios. If we need to track network distance values, we can update it later. In addition, this means bytesReadDistanceOfFour and bytesReadDistanceOfSix won't be used for small network topology. volatile long bytesReadLocalHost; volatile long bytesReadDistanceOfTwo; // local rack case. volatile long bytesReadDistanceOfFour; // first-degree remote rack volatile long bytesReadDistanceOfSix; // second-degree remote rack I will update the patch once we agree on the new approach.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I went over the existing version of the patch.

          First, I don't think using a HashMap for the bytes read per distance is thread safe. Note that one thread (the owner) will modify this map in incrementBytesReadByDistance() while any thread can read the values off the map via getBytesReadByDistance() and visitAll(), all unsynchronized. The problems could range memory visibility, ConcurrentModificationException, and worse. We need to make this thread safe.

          Another reservation I have with using a map: I'm a little concerned about memory implications. An additional map per StatisticsData can add up. Can we find a way of avoiding using a map? I know it may sound ugly, but one other option is to use individual long (volatile) variables. That can also address the thread safety. Thoughts?

          Also, in NetworkTopology.java (lines 373-381) equals() and hashCode() are superfluous here as it does not modify the super behavior in any way.

          Show
          sjlee0 Sangjin Lee added a comment - I went over the existing version of the patch. First, I don't think using a HashMap for the bytes read per distance is thread safe. Note that one thread (the owner) will modify this map in incrementBytesReadByDistance() while any thread can read the values off the map via getBytesReadByDistance() and visitAll() , all unsynchronized. The problems could range memory visibility, ConcurrentModificationException, and worse. We need to make this thread safe. Another reservation I have with using a map: I'm a little concerned about memory implications. An additional map per StatisticsData can add up. Can we find a way of avoiding using a map? I know it may sound ugly, but one other option is to use individual long (volatile) variables. That can also address the thread safety. Thoughts? Also, in NetworkTopology.java (lines 373-381) equals() and hashCode() are superfluous here as it does not modify the super behavior in any way.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Ming Ma, the patch no longer applies cleanly. Do you mind updating the patch? Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Ming Ma , the patch no longer applies cleanly. Do you mind updating the patch? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 7m 40s trunk passed
          -1 compile 4m 57s root in trunk failed with JDK v1.8.0_66.
          -1 compile 5m 37s root in trunk failed with JDK v1.7.0_91.
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 2m 31s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 5m 32s trunk passed
          +1 javadoc 2m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 25s Maven dependency ordering for patch
          +1 mvninstall 2m 52s the patch passed
          -1 compile 0m 36s root in the patch failed with JDK v1.8.0_66.
          -1 javac 0m 36s root in the patch failed with JDK v1.8.0_66.
          -1 compile 5m 36s root in the patch failed with JDK v1.7.0_91.
          -1 javac 5m 36s root in the patch failed with JDK v1.7.0_91.
          -1 checkstyle 1m 4s Patch generated 2 new checkstyle issues in root (total was 407, now 409).
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 6m 8s the patch passed
          +1 javadoc 2m 20s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91
          -1 unit 21m 7s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 66m 4s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 7m 8s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 64m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          222m 12s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestSafeModeWithStripedFile
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782604/HDFS-9579-4.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9939d881bc54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b2c155f
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/branch-compile-root-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/branch-compile-root-jdk1.7.0_91.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.7.0_91.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14136/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14136/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 7m 40s trunk passed -1 compile 4m 57s root in trunk failed with JDK v1.8.0_66. -1 compile 5m 37s root in trunk failed with JDK v1.7.0_91. +1 checkstyle 1m 3s trunk passed +1 mvnsite 2m 31s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 5m 32s trunk passed +1 javadoc 2m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 2m 52s the patch passed -1 compile 0m 36s root in the patch failed with JDK v1.8.0_66. -1 javac 0m 36s root in the patch failed with JDK v1.8.0_66. -1 compile 5m 36s root in the patch failed with JDK v1.7.0_91. -1 javac 5m 36s root in the patch failed with JDK v1.7.0_91. -1 checkstyle 1m 4s Patch generated 2 new checkstyle issues in root (total was 407, now 409). +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 6m 8s the patch passed +1 javadoc 2m 20s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91 -1 unit 21m 7s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 66m 4s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 7m 8s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 64m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 222m 12s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestSafeModeWithStripedFile JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782604/HDFS-9579-4.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9939d881bc54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b2c155f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/branch-compile-root-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/branch-compile-root-jdk1.7.0_91.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-compile-root-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14136/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14136/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14136/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Here is the rebased patch.

          Show
          mingma Ming Ma added a comment - Here is the rebased patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 44s trunk passed
          +1 compile 7m 59s trunk passed with JDK v1.8.0_66
          +1 compile 9m 15s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 2m 29s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 5m 31s trunk passed
          +1 javadoc 2m 24s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 23s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 56s the patch passed
          +1 compile 9m 9s the patch passed with JDK v1.8.0_66
          +1 javac 9m 9s the patch passed
          +1 compile 9m 3s the patch passed with JDK v1.7.0_91
          +1 javac 9m 3s the patch passed
          -1 checkstyle 1m 8s Patch generated 5 new checkstyle issues in root (total was 419, now 422).
          +1 mvnsite 2m 39s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 6m 4s the patch passed
          +1 javadoc 2m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91
          +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 68m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 72m 36s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          239m 35s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779843/HDFS-9579-3.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 04e6c12f9588 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 99cf2ec
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14005/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14005/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 44s trunk passed +1 compile 7m 59s trunk passed with JDK v1.8.0_66 +1 compile 9m 15s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 29s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 5m 31s trunk passed +1 javadoc 2m 24s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 23s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 56s the patch passed +1 compile 9m 9s the patch passed with JDK v1.8.0_66 +1 javac 9m 9s the patch passed +1 compile 9m 3s the patch passed with JDK v1.7.0_91 +1 javac 9m 3s the patch passed -1 checkstyle 1m 8s Patch generated 5 new checkstyle issues in root (total was 419, now 422). +1 mvnsite 2m 39s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 6m 4s the patch passed +1 javadoc 2m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 14s the patch passed with JDK v1.7.0_91 +1 unit 7m 35s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 68m 44s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 59s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 72m 36s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 239m 35s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779843/HDFS-9579-3.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 04e6c12f9588 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 99cf2ec Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14005/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14005/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14005/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated test code to use the actual local host name instead of "localhost".

          Show
          mingma Ming Ma added a comment - Updated test code to use the actual local host name instead of "localhost".
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 47s trunk passed
          +1 compile 7m 59s trunk passed with JDK v1.8.0_66
          +1 compile 8m 40s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 2m 28s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 5m 30s trunk passed
          +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 14s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 57s the patch passed
          +1 compile 7m 37s the patch passed with JDK v1.8.0_66
          +1 javac 7m 37s the patch passed
          +1 compile 8m 44s the patch passed with JDK v1.7.0_91
          +1 javac 8m 44s the patch passed
          -1 checkstyle 1m 4s Patch generated 6 new checkstyle issues in root (total was 380, now 384).
          +1 mvnsite 2m 27s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 58s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues.
          +1 javadoc 2m 18s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 12s the patch passed with JDK v1.7.0_91
          +1 unit 6m 36s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 51m 56s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 6m 39s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 50m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 32s Patch does not generate ASF License warnings.
          194m 14s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.net.NetworkTopology$InnerNode defines equals but not hashCode At NetworkTopology.java:hashCode At NetworkTopology.java:[line 375]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          JDK v1.7.0_91 Failed junit tests hadoop.test.TestTimedOutTestsListener
            hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting
            hadoop.hdfs.TestDistributedFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779696/HDFS-9579-2.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 85cf7bb2d726 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 32b3dc1
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14002/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14002/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 47s trunk passed +1 compile 7m 59s trunk passed with JDK v1.8.0_66 +1 compile 8m 40s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 5m 30s trunk passed +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 14s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 57s the patch passed +1 compile 7m 37s the patch passed with JDK v1.8.0_66 +1 javac 7m 37s the patch passed +1 compile 8m 44s the patch passed with JDK v1.7.0_91 +1 javac 8m 44s the patch passed -1 checkstyle 1m 4s Patch generated 6 new checkstyle issues in root (total was 380, now 384). +1 mvnsite 2m 27s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 58s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues. +1 javadoc 2m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 12s the patch passed with JDK v1.7.0_91 +1 unit 6m 36s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 51m 56s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 6m 39s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 50m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 32s Patch does not generate ASF License warnings. 194m 14s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.net.NetworkTopology$InnerNode defines equals but not hashCode At NetworkTopology.java:hashCode At NetworkTopology.java: [line 375] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.snapshot.TestSnapshot JDK v1.7.0_91 Failed junit tests hadoop.test.TestTimedOutTestsListener   hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting   hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779696/HDFS-9579-2.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85cf7bb2d726 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 32b3dc1 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14002/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14002/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14002/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Updated patch that fixed findbugs and checkstyle.

          Show
          mingma Ming Ma added a comment - Updated patch that fixed findbugs and checkstyle.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 14s trunk passed
          +1 compile 8m 45s trunk passed with JDK v1.8.0_66
          +1 compile 9m 26s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 2m 40s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 5m 56s trunk passed
          +1 javadoc 2m 28s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 25s trunk passed with JDK v1.7.0_91
          +1 mvninstall 3m 8s the patch passed
          +1 compile 8m 44s the patch passed with JDK v1.8.0_66
          +1 javac 8m 44s the patch passed
          +1 compile 9m 24s the patch passed with JDK v1.7.0_91
          +1 javac 9m 24s the patch passed
          -1 checkstyle 1m 2s Patch generated 10 new checkstyle issues in root (total was 381, now 389).
          +1 mvnsite 2m 38s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 8s hadoop-common-project/hadoop-common introduced 2 new FindBugs issues.
          +1 javadoc 2m 27s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 27s the patch passed with JDK v1.7.0_91
          -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 57m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 7m 51s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 53m 57s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 26s Patch generated 1 ASF License warnings.
          211m 24s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Possible null pointer dereference of node1 in org.apache.hadoop.net.NetworkTopology.getDistance(Node, Node) Dereferenced at NetworkTopology.java:node1 in org.apache.hadoop.net.NetworkTopology.getDistance(Node, Node) Dereferenced at NetworkTopology.java:[line 618]
            org.apache.hadoop.net.NetworkTopology$InnerNode doesn't override NodeBase.equals(Object) At NetworkTopology.java:At NetworkTopology.java:[line 1]
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.hdfs.TestDistributedFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778661/HDFS-9579.patch
          JIRA Issue HDFS-9579
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d25475c0b96d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8652cce
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13945/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13945/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 14s trunk passed +1 compile 8m 45s trunk passed with JDK v1.8.0_66 +1 compile 9m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 40s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 5m 56s trunk passed +1 javadoc 2m 28s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 3m 8s the patch passed +1 compile 8m 44s the patch passed with JDK v1.8.0_66 +1 javac 8m 44s the patch passed +1 compile 9m 24s the patch passed with JDK v1.7.0_91 +1 javac 9m 24s the patch passed -1 checkstyle 1m 2s Patch generated 10 new checkstyle issues in root (total was 381, now 389). +1 mvnsite 2m 38s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 8s hadoop-common-project/hadoop-common introduced 2 new FindBugs issues. +1 javadoc 2m 27s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 27s the patch passed with JDK v1.7.0_91 -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 57m 20s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 7m 51s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 53m 57s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 26s Patch generated 1 ASF License warnings. 211m 24s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Possible null pointer dereference of node1 in org.apache.hadoop.net.NetworkTopology.getDistance(Node, Node) Dereferenced at NetworkTopology.java:node1 in org.apache.hadoop.net.NetworkTopology.getDistance(Node, Node) Dereferenced at NetworkTopology.java: [line 618]   org.apache.hadoop.net.NetworkTopology$InnerNode doesn't override NodeBase.equals(Object) At NetworkTopology.java:At NetworkTopology.java: [line 1] JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.hdfs.TestDistributedFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778661/HDFS-9579.patch JIRA Issue HDFS-9579 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d25475c0b96d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8652cce findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13945/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13945/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13945/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          The draft patch has the following changes.

          • Fix NetworkTopology's getDistance to do node comparison not just based on object reference; instead the distance between two {{NodeBase}}s should be zero as long as they have the same network path.
          • Add new metrics to FileSystem.StatisticsData to track bytes read for each distance value.
          • Have DFSInputStream update the new metrics.
          Show
          mingma Ming Ma added a comment - The draft patch has the following changes. Fix NetworkTopology 's getDistance to do node comparison not just based on object reference; instead the distance between two {{NodeBase}}s should be zero as long as they have the same network path. Add new metrics to FileSystem.StatisticsData to track bytes read for each distance value. Have DFSInputStream update the new metrics.

            People

            • Assignee:
              mingma Ming Ma
              Reporter:
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development