Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7163

WebHdfsFileSystem should retry reads according to the configured retry policy.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.1, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: webhdfs
    • Labels:
      None

      Description

      In the current implementation of WebHdfsFileSystem, opens are retried according to the configured retry policy, but not reads. Therefore, if a connection goes down while data is being read, the read will fail and the read will have to be retried by the client code.

      Also, after a connection has been established, the next read (or seek/read) will fail and the read will have to be restarted by the client code.

      1. HDFS-7163.001.patch
        23 kB
        Eric Payne
      2. HDFS-7163.002.patch
        24 kB
        Eric Payne
      3. HDFS-7163.003.patch
        24 kB
        Eric Payne
      4. HDFS-7163.004.patch
        25 kB
        Eric Payne
      5. HDFS-7163.005.patch
        29 kB
        Eric Payne
      6. HDFS-7163-branch-2.003.patch
        23 kB
        Eric Payne
      7. HDFS-7163-branch-2.004.patch
        24 kB
        Eric Payne
      8. HDFS-7163-branch-2.7.003.patch
        23 kB
        Eric Payne
      9. HDFS-7163-branch-2.7.004.patch
        24 kB
        Eric Payne
      10. HDFS-7163-branch-2.7.005.patch
        28 kB
        Eric Payne
      11. WebHDFS Read Retry.pdf
        46 kB
        Eric Payne

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        kihwal Kihwal Lee added a comment -

        The build failure is in libwebhdfs native builds. This is in contrib and only gets built when -Drequire.libwebhdfs is specified. You might need to install some of the build dependencies such as jansson-devel Looks like HDFS-8346.

        Show
        kihwal Kihwal Lee added a comment - The build failure is in libwebhdfs native builds. This is in contrib and only gets built when -Drequire.libwebhdfs is specified. You might need to install some of the build dependencies such as jansson-devel Looks like HDFS-8346 .
        Hide
        eepayne Eric Payne added a comment -
        Vote Subsystem Runtime Comment
        -1 compile 0m 43s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_66.
        -1 compile 0m 44s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91.

        I'm not sure what's wrong with the HDFS pre-commit build. This patch builds for me locally.

        Thanks again, Kihwal Lee.

        Show
        eepayne Eric Payne added a comment - Vote Subsystem Runtime Comment -1 compile 0m 43s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_66. -1 compile 0m 44s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91. I'm not sure what's wrong with the HDFS pre-commit build. This patch builds for me locally. Thanks again, Kihwal Lee .
        Hide
        kihwal Kihwal Lee added a comment -

        +1 The patch looks like a valid port. Committing to branch-2.7.

        Show
        kihwal Kihwal Lee added a comment - +1 The patch looks like a valid port. Committing to branch-2.7.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        +1 mvninstall 10m 51s branch-2.7 passed
        -1 compile 0m 43s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_66.
        -1 compile 0m 44s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91.
        +1 checkstyle 0m 20s branch-2.7 passed
        +1 mvnsite 1m 0s branch-2.7 passed
        +1 mvneclipse 0m 18s branch-2.7 passed
        +1 findbugs 3m 11s branch-2.7 passed
        +1 javadoc 1m 8s branch-2.7 passed with JDK v1.8.0_66
        +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_91
        +1 mvninstall 0m 55s the patch passed
        -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 compile 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 javac 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 checkstyle 0m 17s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 101, now 104).
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 2s The patch has 4418 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 1m 59s The patch has 173 line(s) with tabs.
        +1 findbugs 3m 11s the patch passed
        +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
        -1 unit 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 42m 42s Patch generated 71 ASF License warnings.
        96m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:date2015-12-26
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779554/HDFS-7163-branch-2.7.005.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7fcc3ca898ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / e0e5a0f
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13997/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 68MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13997/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 10m 51s branch-2.7 passed -1 compile 0m 43s hadoop-hdfs in branch-2.7 failed with JDK v1.8.0_66. -1 compile 0m 44s hadoop-hdfs in branch-2.7 failed with JDK v1.7.0_91. +1 checkstyle 0m 20s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 mvneclipse 0m 18s branch-2.7 passed +1 findbugs 3m 11s branch-2.7 passed +1 javadoc 1m 8s branch-2.7 passed with JDK v1.8.0_66 +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_91 +1 mvninstall 0m 55s the patch passed -1 compile 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 javac 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 compile 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 javac 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 checkstyle 0m 17s Patch generated 3 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 101, now 104). +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 2s The patch has 4418 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 59s The patch has 173 line(s) with tabs. +1 findbugs 3m 11s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 0m 43s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 0m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 42m 42s Patch generated 71 ASF License warnings. 96m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-12-26 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779554/HDFS-7163-branch-2.7.005.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7fcc3ca898ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / e0e5a0f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13997/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13997/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 68MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13997/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        I've committed this to trunk, branch-2 and branch-2.8. Eric Payne, please post a 2.7 version.

        Kihwal Lee, Thank you!

        Attaching HDFS-7163-branch-2.7.005.patch

        Show
        eepayne Eric Payne added a comment - I've committed this to trunk, branch-2 and branch-2.8. Eric Payne, please post a 2.7 version. Kihwal Lee , Thank you! Attaching HDFS-7163 -branch-2.7.005.patch
        Hide
        kihwal Kihwal Lee added a comment -

        I've committed this to trunk, branch-2 and branch-2.8. Eric Payne, please post a 2.7 version.

        Show
        kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.8. Eric Payne , please post a 2.7 version.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9015 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9015/)
        HDFS-7163. WebHdfsFileSystem should retry reads according to the (kihwal: rev 867048c3e4b20ece0039a876def129fa5eb9234f)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSXAttrBaseTest.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9015 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9015/ ) HDFS-7163 . WebHdfsFileSystem should retry reads according to the (kihwal: rev 867048c3e4b20ece0039a876def129fa5eb9234f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSXAttrBaseTest.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java
        Hide
        kihwal Kihwal Lee added a comment -

        The latest patch looks good and seems to have addressed all the concerns from the reviews.
        The failed tests pass when run locally.
        +1

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.389 sec - in org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
        Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.679 sec - in org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.477 sec - in org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.388 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.TestDFSClientRetries
        Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.773 sec - in org.apache.hadoop.hdfs.TestDFSClientRetries
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        Running org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
        Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.842 sec - in org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
        
        Results :
        
        Tests run: 36, Failures: 0, Errors: 0, Skipped: 0
        
        Show
        kihwal Kihwal Lee added a comment - The latest patch looks good and seems to have addressed all the concerns from the reviews. The failed tests pass when run locally. +1 ------------------------------------------------------- T E S T S ------------------------------------------------------- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.389 sec - in org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.datanode.TestBlockScanner Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.679 sec - in org.apache.hadoop.hdfs.server.datanode.TestBlockScanner Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.477 sec - in org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.388 sec - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestDFSClientRetries Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.773 sec - in org.apache.hadoop.hdfs.TestDFSClientRetries Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.842 sec - in org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Results : Tests run: 36, Failures: 0, Errors: 0, Skipped: 0
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        -1 mvninstall 7m 32s root in trunk failed.
        -1 compile 1m 26s hadoop-hdfs-project in trunk failed with JDK v1.8.0_66.
        -1 compile 0m 30s hadoop-hdfs-project in trunk failed with JDK v1.7.0_91.
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 25s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 3m 40s trunk passed
        +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 10s trunk passed with JDK v1.7.0_91
        +1 mvninstall 1m 21s the patch passed
        +1 compile 1m 25s the patch passed with JDK v1.8.0_66
        -1 javac 8m 13s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 4 new issues (was 91, now 49).
        +1 javac 1m 25s the patch passed
        +1 compile 1m 33s the patch passed with JDK v1.7.0_91
        -1 javac 9m 46s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 39 new issues (was 14, now 51).
        +1 javac 1m 33s the patch passed
        -1 checkstyle 0m 20s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61).
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 26s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 58s the patch passed
        -1 javadoc 3m 41s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 7, now 14).
        +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66
        -1 javadoc 6m 11s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 7 new issues (was 7, now 14).
        +1 javadoc 2m 8s the patch passed with JDK v1.7.0_91
        +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 52m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 51m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        141m 3s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestBlockReplacement



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0c6902d0df4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 70d6f20
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-compile-hadoop-hdfs-project-jdk1.8.0_66.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-compile-hadoop-hdfs-project-jdk1.7.0_91.txt
        findbugs v3.0.0
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13977/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 74MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13977/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. -1 mvninstall 7m 32s root in trunk failed. -1 compile 1m 26s hadoop-hdfs-project in trunk failed with JDK v1.8.0_66. -1 compile 0m 30s hadoop-hdfs-project in trunk failed with JDK v1.7.0_91. +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 40s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 10s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 21s the patch passed +1 compile 1m 25s the patch passed with JDK v1.8.0_66 -1 javac 8m 13s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 4 new issues (was 91, now 49). +1 javac 1m 25s the patch passed +1 compile 1m 33s the patch passed with JDK v1.7.0_91 -1 javac 9m 46s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 39 new issues (was 14, now 51). +1 javac 1m 33s the patch passed -1 checkstyle 0m 20s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61). +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 58s the patch passed -1 javadoc 3m 41s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new issues (was 7, now 14). +1 javadoc 1m 22s the patch passed with JDK v1.8.0_66 -1 javadoc 6m 11s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 7 new issues (was 7, now 14). +1 javadoc 2m 8s the patch passed with JDK v1.7.0_91 +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 52m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 51m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 141m 3s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0c6902d0df4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 70d6f20 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-compile-hadoop-hdfs-project-jdk1.8.0_66.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/branch-compile-hadoop-hdfs-project-jdk1.7.0_91.txt findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13977/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13977/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 74MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13977/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        Multiple precommit ran on the same build slave, causing them to run more than 5 hours and fail bunch of tests.
        Hopefully it is less busy today.

        Show
        kihwal Kihwal Lee added a comment - Multiple precommit ran on the same build slave, causing them to run more than 5 hours and fail bunch of tests. Hopefully it is less busy today.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        +1 mvninstall 21m 41s trunk passed
        +1 compile 6m 36s trunk passed with JDK v1.8.0_66
        +1 compile 4m 44s trunk passed with JDK v1.7.0_91
        +1 checkstyle 1m 6s trunk passed
        +1 mvnsite 4m 3s trunk passed
        +1 mvneclipse 1m 3s trunk passed
        +1 findbugs 9m 52s trunk passed
        +1 javadoc 4m 52s trunk passed with JDK v1.8.0_66
        +1 javadoc 6m 25s trunk passed with JDK v1.7.0_91
        +1 mvninstall 4m 39s the patch passed
        +1 compile 7m 2s the patch passed with JDK v1.8.0_66
        -1 javac 27m 58s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 49).
        +1 javac 7m 2s the patch passed
        +1 compile 5m 2s the patch passed with JDK v1.7.0_91
        -1 javac 33m 0s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 51).
        +1 javac 5m 2s the patch passed
        -1 checkstyle 1m 6s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61).
        +1 mvnsite 4m 30s the patch passed
        +1 mvneclipse 1m 15s the patch passed
        +1 whitespace 0m 1s Patch has no whitespace issues.
        +1 findbugs 11m 13s the patch passed
        +1 javadoc 4m 56s the patch passed with JDK v1.8.0_66
        +1 javadoc 6m 3s the patch passed with JDK v1.7.0_91
        +1 unit 3m 25s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 229m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 3m 46s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 3m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 1m 31s Patch generated 1 ASF License warnings.
        357m 56s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.server.namenode.TestCheckpoint
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestDFSStorageStateRecovery
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestWriteConfigurationToDFS
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDatanodeDeath
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.datanode.TestFsDatasetCache
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestPersistBlocks
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.TestRecoverStripedFile
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestLocalDFS
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.TestFSImage
          hadoop.hdfs.qjournal.client.TestEpochsAreUnique
          hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.TestDistributedFileSystem
          hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.fs.TestSymlinkHdfsFileContext
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.fs.TestSymlinkHdfsFileSystem
          hadoop.hdfs.qjournal.client.TestQuorumJournalManager
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
          hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDFSInputStream



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1b9cefc264d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bfadf11
        findbugs v3.0.0
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13935/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13935/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 21m 41s trunk passed +1 compile 6m 36s trunk passed with JDK v1.8.0_66 +1 compile 4m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 6s trunk passed +1 mvnsite 4m 3s trunk passed +1 mvneclipse 1m 3s trunk passed +1 findbugs 9m 52s trunk passed +1 javadoc 4m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 6m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 4m 39s the patch passed +1 compile 7m 2s the patch passed with JDK v1.8.0_66 -1 javac 27m 58s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 49). +1 javac 7m 2s the patch passed +1 compile 5m 2s the patch passed with JDK v1.7.0_91 -1 javac 33m 0s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 51). +1 javac 5m 2s the patch passed -1 checkstyle 1m 6s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61). +1 mvnsite 4m 30s the patch passed +1 mvneclipse 1m 15s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 findbugs 11m 13s the patch passed +1 javadoc 4m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 6m 3s the patch passed with JDK v1.7.0_91 +1 unit 3m 25s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 229m 18s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 3m 46s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 3m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 1m 31s Patch generated 1 ASF License warnings. 357m 56s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.TestCheckpoint   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestWriteConfigurationToDFS   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDatanodeDeath   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestLocalDFS   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFSImage   hadoop.hdfs.qjournal.client.TestEpochsAreUnique   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.namenode.TestSecurityTokenEditLog   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.fs.TestSymlinkHdfsFileSystem   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.TestDFSInputStream Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1b9cefc264d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bfadf11 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13935/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13935/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13935/console This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment - - edited

        The precommit didn't compile the code right. It builds fine locally. I will kick the build again.

        /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java:[903,10] error: cannot find symbol
        [ERROR]   symbol:   class WebHdfsInputStream
          location: class TestWebHDFS
        
        Show
        kihwal Kihwal Lee added a comment - - edited The precommit didn't compile the code right. It builds fine locally. I will kick the build again. /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java:[903,10] error: cannot find symbol [ERROR] symbol: class WebHdfsInputStream location: class TestWebHDFS
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        +1 mvninstall 10m 11s trunk passed
        +1 compile 2m 34s trunk passed with JDK v1.8.0_66
        +1 compile 2m 9s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 53s trunk passed
        +1 mvneclipse 0m 31s trunk passed
        +1 findbugs 4m 53s trunk passed
        +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66
        +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91
        +1 mvninstall 2m 3s the patch passed
        +1 compile 2m 43s the patch passed with JDK v1.8.0_66
        -1 javac 12m 45s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 49).
        +1 javac 2m 43s the patch passed
        +1 compile 2m 7s the patch passed with JDK v1.7.0_91
        -1 javac 14m 52s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 51).
        +1 javac 2m 7s the patch passed
        -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61).
        +1 mvnsite 1m 57s the patch passed
        +1 mvneclipse 0m 31s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 5m 18s the patch passed
        +1 javadoc 2m 32s the patch passed with JDK v1.8.0_66
        +1 javadoc 3m 8s the patch passed with JDK v1.7.0_91
        +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 1m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 13s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
        -1 unit 0m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 31s Patch does not generate ASF License warnings.
        58m 8s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4fb7cbcd75a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1de56b0
        findbugs v3.0.0
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13924/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13924/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 10m 11s trunk passed +1 compile 2m 34s trunk passed with JDK v1.8.0_66 +1 compile 2m 9s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 4m 53s trunk passed +1 javadoc 2m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 3s the patch passed +1 compile 2m 43s the patch passed with JDK v1.8.0_66 -1 javac 12m 45s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 48, now 49). +1 javac 2m 43s the patch passed +1 compile 2m 7s the patch passed with JDK v1.7.0_91 -1 javac 14m 52s hadoop-hdfs-project-jdk1.7.0_91 with JDK v1.7.0_91 generated 3 new issues (was 50, now 51). +1 javac 2m 7s the patch passed -1 checkstyle 0m 28s Patch generated 3 new checkstyle issues in hadoop-hdfs-project (total was 58, now 61). +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 18s the patch passed +1 javadoc 2m 32s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 8s the patch passed with JDK v1.7.0_91 +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 1m 2s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 13s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 0m 51s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 58m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778363/HDFS-7163.005.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4fb7cbcd75a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1de56b0 findbugs v3.0.0 javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13924/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13924/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13924/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Daryn Sharp, thank you very much for your in-depth analysis and helpful comments!

        Code:

        1. We may want to defer the open until a read occurs.

        I have done that in this patch, but it caused some of the unit tests to fail because they were expecting the input stream to be open after the fs.open() call. I had to change these tests:

        • FSXAttrBaseTest
        • TestAuditLogs
        • TestWebHdfsFileSystemContract
        • TestWebHdfsTokens

        My question is, are we comfortable that nothing is depending on the current behavior?

        2. runnerState ... should just be initialized as DISCONNECTED.

        Done.

        3. If read(...) throws an IOE due to an explicitly closed stream, will retries occur?

        No. The check for the explicitly closed state happens outside of the retry logic.

        4. In connect(URL), Calling it cachedConnection would clarify its purpose.

        Done.

        5. In getResponse:
        5.1. Should initializeInputStream be unconditionally invoked inside the prior null check on connection? Ie. Is there ever a case when it shouldn't be initialized when a new connection is made?
        5.2. I think the logic should be if (conn != cachedConnection)

        Unknown macro: { cachedConnection = conn; in = initializeInputStream(cachedConnection) }

        If the connection is not cached, initialization always needs to happen. However, the converse is not true. That is, even if connection is cached, initialization still may need to happen.

        For a seek, the connection is cached into cachedConnection by ReadRunner#read after invoking the URLRunner to make the connection. The URLRunner is used rather than the ReadRunner so that AbstractRunner#connect can be told that the URL has already been redirected. On the ohter hand, for a regular read (non-seek case), the ReadRunner#connect makes the connection, but cachedConnection isn't cached until eadRunner#getResponse because we want validateResponse to be run before caching the connection.

        So, in ReadRunner#getResponse, in the seek case, cachedConnection will be non-null, but the input stream (in) will be null. In the regular read case, both will be null.

        So, I took out the check for if cachedConnection is null and always cache it, but I kept the check for if in is null. I realize that cachedConnection doesn't always need to be cached, but the performance cost is small and it makes the code cleaner.

        5.3. Should use URL#getAuthority instead of explicitly extracting and joining the host and port.

        Done.

        6. In ReadRunner#initializeInputStream has a misspelled "performznt".

        Done

        7. In closeInputStream, I'd use IOUtils.closeStream to ensure the close doesn't throw which would prevent the stream state from being updated.

        I replaced in.close() with IOUtils.close(cachedConnection). Is that what you meant?

        8. In general the state management isn't clear. DISCONNECTED vs SEEK appear to be the same, with the exception that SEEK allows the connection to be reopened. When errors occur and the stream is DISCONNECTED, are you sure it will retry/recover in all cases?

        I've done quite a bit of manual testing in a full cluster with reasonably substantial files (16GB). Can you be more specific about your concerns?

        As far as each state is concerned, SEEK and DISCONNECTED are a little different than your comment. Let me try to explain in a little more detail

        • DISCONNECTED
          Connection is closed programmatically by ReadRunner after an exception has occurred. ReadRunner will attempt to open a new connection if it is retried while in this state.
        • OPEN
          Connection has been successfully established by ReadRunner. This occurs after the input stream has been initialized.
        • SEEK
          ReadRunner will only be put in this state if the user code has explicitly called seek(). ReadRunner will use this state as a trigger to perform a redirected connection (as I have discussed above in my reply to your point, 5.1). Once the connection is established and the input stream is initialized, the RunnerState will move to OPEN. Retries will not be attempted while in this state. If an IOException occurs while URLRunner is attempting to open a redirected connection, ReadRunner will move to the DISCOMMECTED state and retry via the normal read path.
        • CLOSED
          ReadRunner is put in this state when user code has explicitly called close().

        Also, as part of this patch, I added a RunnerState parameter to the closeInputStream method. These two are not necessarily tied together, but it does make it clearer (at least in my mind) which state ReadRunner will be moving to as a result of the action. If you are uncomfortable with that, I can separate them out.

        Tests:

        1. In testWebHdfsReadRetries
        1.1. A 5m timeout seems overly generous for something that will hopefully fail much faster if there's a problem.

        Changed to 90 seconds.

        1.2. Why the 5s safemode extension? Seems like it will unnecessarily slow down the test?

        I think that was just from code I copied from the MiniDfsCluster test. I removed it.

        3.The healthy check on dfs appears redundant since cluster#waitActive() already checked.

        I removed it.

        2. The client shouldn't just give up and do nothing for InvalidToken. It's supposed to try and get another token and retry the read. It's unclear if that actually happens or if the position is retaining correctly?

        In a unit test, the client does nothing for InvalidToken exception because security is turned off. I updated the test to mock a token being replaced and then verified that retries happened.

        Kind of a side note: this had to be verified outside of the retry policy verification that I already had mocked up. runWithRetry doesn't use the retry policy when handling InvalidToken exceptions. Instead, it tries to replace the token and, if it succeeds, retries the connection. runWithRetry will retry as many times as that process succeeds with no upper limit. So, I mocked replacing the token twice and then verified that ReadRunner#getResponse was called 3 times.

        1.3. May consider more mockito verifies.

        TestWebHDFS has been enhanced.

        Show
        eepayne Eric Payne added a comment - Daryn Sharp , thank you very much for your in-depth analysis and helpful comments! Code: 1. We may want to defer the open until a read occurs. I have done that in this patch, but it caused some of the unit tests to fail because they were expecting the input stream to be open after the fs.open() call. I had to change these tests: FSXAttrBaseTest TestAuditLogs TestWebHdfsFileSystemContract TestWebHdfsTokens My question is, are we comfortable that nothing is depending on the current behavior? 2. runnerState ... should just be initialized as DISCONNECTED. Done. 3. If read(...) throws an IOE due to an explicitly closed stream, will retries occur? No. The check for the explicitly closed state happens outside of the retry logic. 4. In connect(URL) , Calling it cachedConnection would clarify its purpose. Done. 5. In getResponse: 5.1. Should initializeInputStream be unconditionally invoked inside the prior null check on connection? Ie. Is there ever a case when it shouldn't be initialized when a new connection is made? 5.2. I think the logic should be if (conn != cachedConnection) Unknown macro: { cachedConnection = conn; in = initializeInputStream(cachedConnection) } If the connection is not cached, initialization always needs to happen. However, the converse is not true. That is, even if connection is cached, initialization still may need to happen. For a seek, the connection is cached into cachedConnection by ReadRunner#read after invoking the URLRunner to make the connection. The URLRunner is used rather than the ReadRunner so that AbstractRunner#connect can be told that the URL has already been redirected. On the ohter hand, for a regular read (non-seek case), the ReadRunner#connect makes the connection, but cachedConnection isn't cached until eadRunner#getResponse because we want validateResponse to be run before caching the connection. So, in ReadRunner#getResponse , in the seek case, cachedConnection will be non-null, but the input stream ( in ) will be null. In the regular read case, both will be null. So, I took out the check for if cachedConnection is null and always cache it, but I kept the check for if in is null. I realize that cachedConnection doesn't always need to be cached, but the performance cost is small and it makes the code cleaner. 5.3. Should use URL#getAuthority instead of explicitly extracting and joining the host and port. Done. 6. In ReadRunner#initializeInputStream has a misspelled "performznt". Done 7. In closeInputStream , I'd use IOUtils.closeStream to ensure the close doesn't throw which would prevent the stream state from being updated. I replaced in.close() with IOUtils.close(cachedConnection) . Is that what you meant? 8. In general the state management isn't clear. DISCONNECTED vs SEEK appear to be the same, with the exception that SEEK allows the connection to be reopened. When errors occur and the stream is DISCONNECTED, are you sure it will retry/recover in all cases? I've done quite a bit of manual testing in a full cluster with reasonably substantial files (16GB). Can you be more specific about your concerns? As far as each state is concerned, SEEK and DISCONNECTED are a little different than your comment. Let me try to explain in a little more detail DISCONNECTED Connection is closed programmatically by ReadRunner after an exception has occurred. ReadRunner will attempt to open a new connection if it is retried while in this state. OPEN Connection has been successfully established by ReadRunner . This occurs after the input stream has been initialized. SEEK ReadRunner will only be put in this state if the user code has explicitly called seek(). ReadRunner will use this state as a trigger to perform a redirected connection (as I have discussed above in my reply to your point, 5.1). Once the connection is established and the input stream is initialized, the RunnerState will move to OPEN. Retries will not be attempted while in this state. If an IOException occurs while URLRunner is attempting to open a redirected connection, ReadRunner will move to the DISCOMMECTED state and retry via the normal read path. CLOSED ReadRunner is put in this state when user code has explicitly called close(). Also, as part of this patch, I added a RunnerState parameter to the closeInputStream method. These two are not necessarily tied together, but it does make it clearer (at least in my mind) which state ReadRunner will be moving to as a result of the action. If you are uncomfortable with that, I can separate them out. Tests: 1. In testWebHdfsReadRetries 1.1. A 5m timeout seems overly generous for something that will hopefully fail much faster if there's a problem. Changed to 90 seconds. 1.2. Why the 5s safemode extension? Seems like it will unnecessarily slow down the test? I think that was just from code I copied from the MiniDfsCluster test. I removed it. 3.The healthy check on dfs appears redundant since cluster#waitActive() already checked. I removed it. 2. The client shouldn't just give up and do nothing for InvalidToken. It's supposed to try and get another token and retry the read. It's unclear if that actually happens or if the position is retaining correctly? In a unit test, the client does nothing for InvalidToken exception because security is turned off. I updated the test to mock a token being replaced and then verified that retries happened. Kind of a side note: this had to be verified outside of the retry policy verification that I already had mocked up. runWithRetry doesn't use the retry policy when handling InvalidToken exceptions. Instead, it tries to replace the token and, if it succeeds, retries the connection. runWithRetry will retry as many times as that process succeeds with no upper limit. So, I mocked replacing the token twice and then verified that ReadRunner#getResponse was called 3 times. 1.3. May consider more mockito verifies. TestWebHDFS has been enhanced.
        Hide
        daryn Daryn Sharp added a comment -

        Regarding use of client-side block locations, I did that internally for 0.23 and it wasn't as easy as it seemed. The excluded nodes support is actually much cleaner for the client. The NN also issues the redirect based on the block location of the offset.

        Code:

        1. We may want to defer the open until a read occurs. Otherwise when immediately seeking (ex. splits), the file will start unnecessarily streaming and consuming bandwidth, the client will close the connection, then reopen at the seek offset.
        2. runnerState is initialized as CLOSED (terminal state) although reassigned as DISCONNECTED in the ctor. Feels like it should just be initialized as DISCONNECTED.
        3. If read(...) throws an IOE due to an explicitly closed stream, will retries occur?
        4. In connect(URL), conn = connection, conn is re-assigned if null, conn is returned, but connection remains null. This seems confusing and wrong at first. Calling it cachedConnection would clarify its purpose.
        5. In getResponse:
          1. Should initializeInputStream be unconditionally invoked inside the prior null check on connection? Ie. Is there ever a case when in shouldn't be initialized when a new connection is made?
          2. I think the logic should be {{if (conn != cachedConnection) { cachedConnection = conn; in = initializeInputStream(cachedConnection) }

            }}

          3. Should use URL#getAuthority instead of explicitly extracting and joining the host and port.
        6. In ReadRunner#initializeInputStream has a misspelled "performznt".
        7. In closeInputStream, I'd use IOUtils.closeStream to ensure the close doesn't throw which would prevent the stream state from being updated.
        8. In general the state management isn't clear. DISCONNECTED vs SEEK appear to be the same, with the exception that SEEK allows the connection to be reopened. When errors occur and the stream is DISCONNECTED, are you sure it will retry/recover in all cases?

        Tests:

        1. In testWebHdfsReadRetries
          1. A 5m timeout seems overly generous for something that will hopefully fail much faster if there's a problem.
          2. Why the 5s safemode extension? Seems like it will unnecessarily slow down the test?
          3. The healthy check on dfs appears redundant since cluster#waitActive() already checked.
        2. The client shouldn't just give up and do nothing for InvalidToken. It's supposed to try and get another token and retry the read. It's unclear if that actually happens or if the position is retaining correctly?
        3. May consider more mockito verifies.
        Show
        daryn Daryn Sharp added a comment - Regarding use of client-side block locations, I did that internally for 0.23 and it wasn't as easy as it seemed. The excluded nodes support is actually much cleaner for the client. The NN also issues the redirect based on the block location of the offset. Code: We may want to defer the open until a read occurs. Otherwise when immediately seeking (ex. splits), the file will start unnecessarily streaming and consuming bandwidth, the client will close the connection, then reopen at the seek offset. runnerState is initialized as CLOSED (terminal state) although reassigned as DISCONNECTED in the ctor. Feels like it should just be initialized as DISCONNECTED. If read(...) throws an IOE due to an explicitly closed stream, will retries occur? In connect(URL) , conn = connection , conn is re-assigned if null, conn is returned, but connection remains null. This seems confusing and wrong at first. Calling it cachedConnection would clarify its purpose. In getResponse : Should initializeInputStream be unconditionally invoked inside the prior null check on connection? Ie. Is there ever a case when in shouldn't be initialized when a new connection is made? I think the logic should be {{if (conn != cachedConnection) { cachedConnection = conn; in = initializeInputStream(cachedConnection) } }} Should use URL#getAuthority instead of explicitly extracting and joining the host and port. In ReadRunner#initializeInputStream has a misspelled "performznt". In closeInputStream , I'd use IOUtils.closeStream to ensure the close doesn't throw which would prevent the stream state from being updated. In general the state management isn't clear. DISCONNECTED vs SEEK appear to be the same, with the exception that SEEK allows the connection to be reopened. When errors occur and the stream is DISCONNECTED, are you sure it will retry/recover in all cases? Tests: In testWebHdfsReadRetries A 5m timeout seems overly generous for something that will hopefully fail much faster if there's a problem. Why the 5s safemode extension? Seems like it will unnecessarily slow down the test? The healthy check on dfs appears redundant since cluster#waitActive() already checked. The client shouldn't just give up and do nothing for InvalidToken. It's supposed to try and get another token and retry the read. It's unclear if that actually happens or if the position is retaining correctly? May consider more mockito verifies.
        Hide
        eepayne Eric Payne added a comment -

        Haohui Mai and Daryn Sharp,
        Did my comments above make sense:

        In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred.

        An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations.

        Although this may be a more elegant solution, I think that could be done as part of a separate JIRA, given that we can take advantage of the exclude list functionality as I mentioned above.

        Show
        eepayne Eric Payne added a comment - Haohui Mai and Daryn Sharp , Did my comments above make sense: In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred. An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations. Although this may be a more elegant solution, I think that could be done as part of a separate JIRA, given that we can take advantage of the exclude list functionality as I mentioned above.
        Hide
        eepayne Eric Payne added a comment -

        Looks like TestPermission#testBackwardCompatibility is broken by HADOOP-12294 as documented here

        Show
        eepayne Eric Payne added a comment - Looks like TestPermission#testBackwardCompatibility is broken by HADOOP-12294 as documented here
        Hide
        eepayne Eric Payne added a comment -

        Although the following tests are listed in the above Failed unit tests section, they all passed for me in my local environment:

        hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
        hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
        hadoop.hdfs.server.datanode.TestDataNodeMetrics
        hadoop.hdfs.server.namenode.ha.TestHASafeMode
        hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
        hadoop.hdfs.server.namenode.TestDecommissioningStatus
        hadoop.hdfs.shortcircuit.TestShortCircuitCache
        hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
        hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
        hadoop.hdfs.TestEncryptionZones
        hadoop.hdfs.TestReadStripedFileWithDecoding
        hadoop.hdfs.TestReplaceDatanodeOnFailure
        hadoop.hdfs.web.TestWebHDFS
        

        And, this one failed with and without my patch:

        hadoop.security.TestPermission
        
        Show
        eepayne Eric Payne added a comment - Although the following tests are listed in the above Failed unit tests section, they all passed for me in my local environment: hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation hadoop.hdfs.server.datanode.TestDataNodeMetrics hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.namenode.TestDecommissioningStatus hadoop.hdfs.shortcircuit.TestShortCircuitCache hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 hadoop.hdfs.TestEncryptionZones hadoop.hdfs.TestReadStripedFileWithDecoding hadoop.hdfs.TestReplaceDatanodeOnFailure hadoop.hdfs.web.TestWebHDFS And, this one failed with and without my patch: hadoop.security.TestPermission
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 12s trunk passed
        +1 compile 1m 37s trunk passed with JDK v1.8.0_66
        +1 compile 1m 41s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 1m 36s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 4m 15s trunk passed
        +1 javadoc 1m 44s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 36s trunk passed with JDK v1.7.0_85
        -1 mvninstall 0m 45s hadoop-hdfs in the patch failed.
        +1 compile 1m 42s the patch passed with JDK v1.8.0_66
        -1 javac 9m 35s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 50).
        +1 javac 1m 42s the patch passed
        +1 compile 1m 43s the patch passed with JDK v1.7.0_85
        -1 javac 11m 18s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 52).
        +1 javac 1m 43s the patch passed
        -1 checkstyle 0m 24s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 58, now 59).
        +1 mvnsite 1m 35s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 37s the patch passed
        +1 javadoc 1m 38s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 23s the patch passed with JDK v1.7.0_85
        -1 unit 66m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 57m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
        +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
        -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
        167m 52s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.security.TestPermission
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
        JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.security.TestPermission
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestEncryptionZones



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774080/HDFS-7163.004.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 807287c61b61 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 28dfe72
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
        javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13633/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13633/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 12s trunk passed +1 compile 1m 37s trunk passed with JDK v1.8.0_66 +1 compile 1m 41s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 4m 15s trunk passed +1 javadoc 1m 44s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 36s trunk passed with JDK v1.7.0_85 -1 mvninstall 0m 45s hadoop-hdfs in the patch failed. +1 compile 1m 42s the patch passed with JDK v1.8.0_66 -1 javac 9m 35s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 3 new issues (was 49, now 50). +1 javac 1m 42s the patch passed +1 compile 1m 43s the patch passed with JDK v1.7.0_85 -1 javac 11m 18s hadoop-hdfs-project-jdk1.7.0_85 with JDK v1.7.0_85 generated 3 new issues (was 51, now 52). +1 javac 1m 43s the patch passed -1 checkstyle 0m 24s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 58, now 59). +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 37s the patch passed +1 javadoc 1m 38s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 23s the patch passed with JDK v1.7.0_85 -1 unit 66m 11s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 57m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_85. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 167m 52s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.security.TestPermission   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.security.TestPermission   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774080/HDFS-7163.004.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 807287c61b61 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28dfe72 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13633/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13633/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13633/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Removed and reattached HDFS-7163.004.patch in hopes of re-launching the precommit build.

        https://builds.apache.org/job/PreCommit-HDFS-Build/13627/ ran, but did not complete.

        Show
        eepayne Eric Payne added a comment - Removed and reattached HDFS-7163 .004.patch in hopes of re-launching the precommit build. https://builds.apache.org/job/PreCommit-HDFS-Build/13627/ ran, but did not complete.
        Hide
        eepayne Eric Payne added a comment -

        Attaching version 004 of patches for trunk, branch-2, and branch-2.7.

        Version 003 had the issue that could result in the NN sending the client to a bad DN one extra time. In version 003, if the client received an IOException while reading from the DN, it failed to put the DN in the excluded nodes list. This could result in the NN sending the client back to the same DN. However, if that occurred, the open would fail and send the client back to the NN, this time with the bad DN in the excluded nodes list. The read would still succeed, but it would take a bit longer due to an extra attempt to open a bad DN.

        Version 004 fixes that issue and supplies the bad DN in the excluded nodes list during a read when an IOException occurs.

        Show
        eepayne Eric Payne added a comment - Attaching version 004 of patches for trunk, branch-2, and branch-2.7. Version 003 had the issue that could result in the NN sending the client to a bad DN one extra time. In version 003, if the client received an IOException while reading from the DN, it failed to put the DN in the excluded nodes list. This could result in the NN sending the client back to the same DN. However, if that occurred, the open would fail and send the client back to the NN, this time with the bad DN in the excluded nodes list. The read would still succeed, but it would take a bit longer due to an extra attempt to open a bad DN. Version 004 fixes that issue and supplies the bad DN in the excluded nodes list during a read when an IOException occurs.
        Hide
        eepayne Eric Payne added a comment -

        In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred.

        Haohui Mai, I was wrong about one thing. In the current patch, a failed read does not put the current node into the exclude list, so when the client queries the NN again, the NN could give it the same node back. I will put up a new patch soon addressing this issue.

        Show
        eepayne Eric Payne added a comment - In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred. Haohui Mai , I was wrong about one thing. In the current patch, a failed read does not put the current node into the exclude list, so when the client queries the NN again, the NN could give it the same node back. I will put up a new patch soon addressing this issue.
        Hide
        eepayne Eric Payne added a comment -

        Haohui Mai, thank you for your review and comments on this feature.

        I think retrying only on the data node is problematic as the retry might have little value when the DN goes down.

        In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred.

        An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations.

        Although this may be a more elegant solution, I think that could be done as part of a separate JIRA, given that we can take advantage of the exclude list functionality as I mentioned above.

        Show
        eepayne Eric Payne added a comment - Haohui Mai , thank you for your review and comments on this feature. I think retrying only on the data node is problematic as the retry might have little value when the DN goes down. In this patch, if the DN that is being read from goes down, WebHDFS will put that DN into the client's URL exclude list before querying the NN again for another DN. The only time the same DN is reused is if a seek has occurred. An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations. Although this may be a more elegant solution, I think that could be done as part of a separate JIRA, given that we can take advantage of the exclude list functionality as I mentioned above.
        Hide
        wheat9 Haohui Mai added a comment -

        I think retrying only on the data node is problematic as the retry might have little value when the DN goes down.

        An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations.

        Show
        wheat9 Haohui Mai added a comment - I think retrying only on the data node is problematic as the retry might have little value when the DN goes down. An alternative approach is to have WebHDFS (1) expose a GET_BLOCK call where the DN returns the block directly, and (2) be a smarter client that retries based on block locations.
        Hide
        eepayne Eric Payne added a comment -

        Hi Haohui Mai and Daryn Sharp. I wonder if either of you know when you might have a chance to review this. I would really appreciate your feedback. Thanks.

        Show
        eepayne Eric Payne added a comment - Hi Haohui Mai and Daryn Sharp . I wonder if either of you know when you might have a chance to review this. I would really appreciate your feedback. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 16s branch-2.7 passed
        +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_66
        +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_79
        +1 checkstyle 0m 24s branch-2.7 passed
        +1 mvnsite 1m 0s branch-2.7 passed
        +1 mvneclipse 0m 21s branch-2.7 passed
        +1 findbugs 3m 46s branch-2.7 passed
        +1 javadoc 1m 34s branch-2.7 passed with JDK v1.8.0_66
        +1 javadoc 2m 22s branch-2.7 passed with JDK v1.7.0_79
        +1 mvninstall 0m 58s the patch passed
        +1 compile 1m 8s the patch passed with JDK v1.8.0_66
        +1 javac 1m 8s the patch passed
        +1 compile 1m 1s the patch passed with JDK v1.7.0_79
        +1 javac 1m 1s the patch passed
        -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 101, now 102).
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        -1 whitespace 0m 2s The patch has 2114 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 56s The patch has 95 line(s) with tabs.
        +1 findbugs 3m 50s the patch passed
        +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 22s the patch passed with JDK v1.7.0_79
        -1 unit 61m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 60m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 47m 44s Patch generated 72 ASF License warnings.
        205m 58s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.TestFileTruncate
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockReaderLocal
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestAppendDifferentChecksum



        Subsystem Report/Notes
        Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-12
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771998/HDFS-7163-branch-2.7.003.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4635e20beab4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
        git revision branch-2.7 / adede3e
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13488/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 207MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13488/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 16s branch-2.7 passed +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_66 +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_79 +1 checkstyle 0m 24s branch-2.7 passed +1 mvnsite 1m 0s branch-2.7 passed +1 mvneclipse 0m 21s branch-2.7 passed +1 findbugs 3m 46s branch-2.7 passed +1 javadoc 1m 34s branch-2.7 passed with JDK v1.8.0_66 +1 javadoc 2m 22s branch-2.7 passed with JDK v1.7.0_79 +1 mvninstall 0m 58s the patch passed +1 compile 1m 8s the patch passed with JDK v1.8.0_66 +1 javac 1m 8s the patch passed +1 compile 1m 1s the patch passed with JDK v1.7.0_79 +1 javac 1m 1s the patch passed -1 checkstyle 0m 22s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 101, now 102). +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 16s the patch passed -1 whitespace 0m 2s The patch has 2114 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 56s The patch has 95 line(s) with tabs. +1 findbugs 3m 50s the patch passed +1 javadoc 1m 33s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 22s the patch passed with JDK v1.7.0_79 -1 unit 61m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 60m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 47m 44s Patch generated 72 ASF License warnings. 205m 58s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.TestFileTruncate JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestAppendDifferentChecksum Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-12 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771998/HDFS-7163-branch-2.7.003.patch JIRA Issue HDFS-7163 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4635e20beab4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision branch-2.7 / adede3e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13488/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13488/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 207MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13488/console This message was automatically generated.
        Hide
        wheat9 Haohui Mai added a comment -

        Sure. Will look into it next week.

        Show
        wheat9 Haohui Mai added a comment - Sure. Will look into it next week.
        Hide
        eepayne Eric Payne added a comment -

        As documented above, the unit test errors are not occurring for me in my local build environment.

        Attaching branch-2 and branch-2.7 patches. Although I named them according to the naming convention documented here, the build will still try to apply them to trunk, so the corresponding HadoopQA message will indicate a build failure.

        Haohui Mai, Daryn Sharp, can you please take a look at this patch? Thank you.

        Show
        eepayne Eric Payne added a comment - As documented above, the unit test errors are not occurring for me in my local build environment. Attaching branch-2 and branch-2.7 patches. Although I named them according to the naming convention documented here , the build will still try to apply them to trunk, so the corresponding HadoopQA message will indicate a build failure. Haohui Mai , Daryn Sharp , can you please take a look at this patch? Thank you.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 5s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 3m 1s trunk passed
        +1 compile 0m 58s trunk passed with JDK v1.8.0_60
        +1 compile 0m 56s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 19s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 25s trunk passed with JDK v1.8.0_60
        +1 javadoc 2m 9s trunk passed with JDK v1.7.0_79
        -1 mvninstall 0m 29s hadoop-hdfs in the patch failed.
        +1 compile 0m 57s the patch passed with JDK v1.8.0_60
        +1 javac 0m 57s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.7.0_79
        +1 javac 0m 55s the patch passed
        -1 checkstyle 0m 20s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 58, now 59).
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 1m 25s the patch passed with JDK v1.8.0_60
        +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79
        -1 unit 50m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60.
        +1 unit 48m 41s hadoop-hdfs in the patch passed with JDK v1.7.0_79.
        +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        128m 17s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-06
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771091/HDFS-7163.003.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux dda9e18f9201 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
        git revision trunk / bf6aa30
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13421/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 225MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13421/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 1s trunk passed +1 compile 0m 58s trunk passed with JDK v1.8.0_60 +1 compile 0m 56s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 19s trunk passed +1 mvneclipse 0m 25s trunk passed -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 25s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 9s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 29s hadoop-hdfs in the patch failed. +1 compile 0m 57s the patch passed with JDK v1.8.0_60 +1 javac 0m 57s the patch passed +1 compile 0m 55s the patch passed with JDK v1.7.0_79 +1 javac 0m 55s the patch passed -1 checkstyle 0m 20s Patch generated 1 new checkstyle issues in hadoop-hdfs-project (total was 58, now 59). +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 57s the patch passed +1 javadoc 1m 25s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79 -1 unit 50m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60. +1 unit 48m 41s hadoop-hdfs in the patch passed with JDK v1.7.0_79. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 128m 17s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-06 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771091/HDFS-7163.003.patch JIRA Issue HDFS-7163 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux dda9e18f9201 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / bf6aa30 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13421/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13421/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13421/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Fixed the checkstyle and findbugs warnings. None of the unit tests listed above failed in my own build environment.

        Show
        eepayne Eric Payne added a comment - Fixed the checkstyle and findbugs warnings. None of the unit tests listed above failed in my own build environment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 3m 16s trunk passed
        +1 compile 1m 10s trunk passed with JDK v1.8.0_66
        +1 compile 1m 4s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 21s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 2 extant Findbugs warnings.
        +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66
        +1 javadoc 2m 29s trunk passed with JDK v1.7.0_79
        -1 mvninstall 0m 34s hadoop-hdfs in the patch failed.
        +1 compile 1m 4s the patch passed with JDK v1.8.0_66
        +1 javac 1m 4s the patch passed
        +1 compile 1m 1s the patch passed with JDK v1.7.0_79
        +1 javac 1m 1s the patch passed
        -1 checkstyle 0m 23s Patch generated 7 new checkstyle issues in hadoop-hdfs-project (total was 58, now 65).
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues.
        +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66
        +1 javadoc 2m 29s the patch passed with JDK v1.7.0_79
        -1 unit 71m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
        -1 unit 71m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        +1 unit 1m 5s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
        -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
        175m 15s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
          org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getResponse(HttpURLConnection) invokes inefficient new Integer(int) constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:[line 1781]
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.TestLease



        Subsystem Report/Notes
        Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-05
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770868/HDFS-7163.002.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux 844e68efd082 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
        git revision trunk / 21c0e3e
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13406/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 226MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13406/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 16s trunk passed +1 compile 1m 10s trunk passed with JDK v1.8.0_66 +1 compile 1m 4s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 21s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 2 extant Findbugs warnings. +1 javadoc 1m 38s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 29s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 34s hadoop-hdfs in the patch failed. +1 compile 1m 4s the patch passed with JDK v1.8.0_66 +1 javac 1m 4s the patch passed +1 compile 1m 1s the patch passed with JDK v1.7.0_79 +1 javac 1m 1s the patch passed -1 checkstyle 0m 23s Patch generated 7 new checkstyle issues in hadoop-hdfs-project (total was 58, now 65). +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 14s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues. +1 javadoc 1m 41s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 29s the patch passed with JDK v1.7.0_79 -1 unit 71m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 71m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 1m 5s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 175m 15s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getResponse(HttpURLConnection) invokes inefficient new Integer(int) constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java: [line 1781] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.TestLease Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-05 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770868/HDFS-7163.002.patch JIRA Issue HDFS-7163 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 844e68efd082 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 21c0e3e Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13406/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13406/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13406/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        New patch (HDFS-7163.002.patch). Fixed unit test failures for TestWriteReadStripedFile. Also fixed javadoc and whitespace warnings.

        The following tests did not fail for me in my build environment, so I don't think they are related:
        hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
        hadoop.hdfs.server.blockmanagement.TestNodeCount
        hadoop.hdfs.server.namenode.ha.TestDNFencing
        hadoop.hdfs.server.namenode.TestCacheDirectives
        hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000

        I will shortly post branch-2 and branch-2.7 patches.

        Show
        eepayne Eric Payne added a comment - New patch ( HDFS-7163 .002.patch). Fixed unit test failures for TestWriteReadStripedFile . Also fixed javadoc and whitespace warnings. The following tests did not fail for me in my build environment, so I don't think they are related: hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes hadoop.hdfs.server.blockmanagement.TestNodeCount hadoop.hdfs.server.namenode.ha.TestDNFencing hadoop.hdfs.server.namenode.TestCacheDirectives hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 I will shortly post branch-2 and branch-2.7 patches.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 5s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 13s trunk passed
        +1 compile 0m 57s trunk passed with JDK v1.8.0_60
        +1 compile 0m 54s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 19s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
        +1 javadoc 1m 25s trunk passed with JDK v1.8.0_60
        +1 javadoc 2m 6s trunk passed with JDK v1.7.0_79
        -1 mvninstall 0m 30s hadoop-hdfs in the patch failed.
        +1 compile 0m 54s the patch passed with JDK v1.8.0_60
        +1 javac 0m 54s the patch passed
        +1 compile 0m 54s the patch passed with JDK v1.7.0_79
        +1 javac 0m 54s the patch passed
        -1 checkstyle 0m 19s Patch generated 7 new checkstyle issues in hadoop-hdfs-project (total was 58, now 65).
        +1 mvneclipse 0m 25s the patch passed
        -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues.
        -1 javadoc 3m 45s hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60 with JDK v1.8.0_60 has problems.
        +1 javadoc 1m 27s the patch passed with JDK v1.8.0_60
        -1 javadoc 6m 15s hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79 with JDK v1.7.0_79 has problems.
        +1 javadoc 2m 9s the patch passed with JDK v1.7.0_79
        -1 unit 54m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        +1 unit 0m 26s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60.
        -1 unit 53m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        +1 unit 0m 26s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        135m 21s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client
          org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getResponse(HttpURLConnection) invokes inefficient new Integer(int) constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:[line 1770]
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
        JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769358/HDFS-7163.001.patch
        JIRA Issue HDFS-7163
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux dc3bf0e11e18 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
        git revision trunk / 27414da
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html
        javadoc hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60-diff.txt
        javadoc hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79-diff.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13258/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
        Max memory used 225MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13258/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 13s trunk passed +1 compile 0m 57s trunk passed with JDK v1.8.0_60 +1 compile 0m 54s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 19s trunk passed +1 mvneclipse 0m 25s trunk passed -1 findbugs 1m 54s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 25s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 6s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 30s hadoop-hdfs in the patch failed. +1 compile 0m 54s the patch passed with JDK v1.8.0_60 +1 javac 0m 54s the patch passed +1 compile 0m 54s the patch passed with JDK v1.7.0_79 +1 javac 0m 54s the patch passed -1 checkstyle 0m 19s Patch generated 7 new checkstyle issues in hadoop-hdfs-project (total was 58, now 65). +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs-client introduced 1 new FindBugs issues. -1 javadoc 3m 45s hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60 with JDK v1.8.0_60 has problems. +1 javadoc 1m 27s the patch passed with JDK v1.8.0_60 -1 javadoc 6m 15s hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79 with JDK v1.7.0_79 has problems. +1 javadoc 2m 9s the patch passed with JDK v1.7.0_79 -1 unit 54m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 0m 26s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60. -1 unit 53m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 0m 26s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 135m 21s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs-client   org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getResponse(HttpURLConnection) invokes inefficient new Integer(int) constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java:constructor; use Integer.valueOf(int) instead At WebHdfsFileSystem.java: [line 1770] JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769358/HDFS-7163.001.patch JIRA Issue HDFS-7163 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux dc3bf0e11e18 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh git revision trunk / 27414da Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.html javadoc hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.8.0_60-diff.txt javadoc hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdk1.7.0_79-diff.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13258/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13258/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13258/console This message was automatically generated.
        Hide
        eepayne Eric Payne added a comment -

        Daryn Sharp, Haohui Mai, Kihwal Lee. Please find attached the design and patch for adding read retry support to WebHdfs. I would really appreciate your input and feedback.

        Show
        eepayne Eric Payne added a comment - Daryn Sharp , Haohui Mai , Kihwal Lee . Please find attached the design and patch for adding read retry support to WebHdfs. I would really appreciate your input and feedback.

          People

          • Assignee:
            eepayne Eric Payne
            Reporter:
            eepayne Eric Payne
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development