Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11195

Return error when appending files by webhdfs rest api fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Suppose that there is a Hadoop cluster contains only one datanode, and dfs.replication=3. Run:

      curl -i -X POST -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND"
      

      it returns 200, even though append operation fails.

      1. HDFS-11195.001.patch
        8 kB
        Yuanbo Liu
      2. HDFS-11195.002.patch
        5 kB
        Yuanbo Liu
      3. HDFS-11195.003.patch
        5 kB
        Yuanbo Liu
      4. HDFS-11195.004.patch
        5 kB
        Yuanbo Liu

        Activity

        Hide
        yuanbo Yuanbo Liu added a comment -

        I have reproduced this issue in trunk branch, but still doesn't come up with a good solution to solve this problem.
        The related code is here WebHdfsHandler#onAppend:

          private void onAppend(ChannelHandlerContext ctx) throws IOException {
             ....
            resp = new DefaultHttpResponse(HTTP_1_1, OK);
            resp.headers().set(CONTENT_LENGTH, 0);
            ctx.pipeline().replace(this, HdfsWriter.class.getSimpleName(),
              new HdfsWriter(dfsClient, out, resp));
          }
        

        HdfsWriter doesn't write successfully, but the response has been set to 200(OK).

        Show
        yuanbo Yuanbo Liu added a comment - I have reproduced this issue in trunk branch, but still doesn't come up with a good solution to solve this problem. The related code is here WebHdfsHandler#onAppend : private void onAppend(ChannelHandlerContext ctx) throws IOException { .... resp = new DefaultHttpResponse(HTTP_1_1, OK); resp.headers().set(CONTENT_LENGTH, 0); ctx.pipeline().replace( this , HdfsWriter.class.getSimpleName(), new HdfsWriter(dfsClient, out, resp)); } HdfsWriter doesn't write successfully, but the response has been set to 200(OK).
        Hide
        yuanbo Yuanbo Liu added a comment -

        upload v1 patch for this issue.

        Show
        yuanbo Yuanbo Liu added a comment - upload v1 patch for this issue.
        Hide
        yuanbo Yuanbo Liu added a comment -

        Xiao Chen / Wei-Chiu Chuang would you mind taking a look at my patch? thanks in advance.

        Show
        yuanbo Yuanbo Liu added a comment - Xiao Chen / Wei-Chiu Chuang would you mind taking a look at my patch? thanks in advance.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 24s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 57s trunk passed
        +1 javadoc 0m 43s trunk passed
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 47s the patch passed
        +1 javac 0m 47s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 250 unchanged - 1 fixed = 252 total (was 251)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
        +1 javadoc 0m 42s the patch passed
        -1 unit 67m 32s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        87m 52s



        Reason Tests
        FindBugs module:hadoop-hdfs-project/hadoop-hdfs
          org.apache.hadoop.hdfs.server.datanode.web.webhdfs.HdfsWriter.channelInactive(ChannelHandlerContext) might ignore java.lang.Exception At HdfsWriter.java:At HdfsWriter.java:[line 73]
          org.apache.hadoop.hdfs.server.datanode.web.webhdfs.HdfsWriter.exceptionCaught(ChannelHandlerContext, Throwable) might ignore java.lang.Exception At HdfsWriter.java: At HdfsWriter.java:[line 80]
        Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.TestSecureEncryptionZoneWithKMS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11195
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842707/HDFS-11195.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6b91afec8dac 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4c38f11
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17829/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17829/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 250 unchanged - 1 fixed = 252 total (was 251) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 42s the patch passed -1 unit 67m 32s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 87m 52s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   org.apache.hadoop.hdfs.server.datanode.web.webhdfs.HdfsWriter.channelInactive(ChannelHandlerContext) might ignore java.lang.Exception At HdfsWriter.java:At HdfsWriter.java: [line 73]   org.apache.hadoop.hdfs.server.datanode.web.webhdfs.HdfsWriter.exceptionCaught(ChannelHandlerContext, Throwable) might ignore java.lang.Exception At HdfsWriter.java: At HdfsWriter.java: [line 80] Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestSecureEncryptionZoneWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842707/HDFS-11195.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b91afec8dac 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/17829/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17829/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17829/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Yuanbo Liu for reporting the issue and providing a fix.

        Could you explain a little more about your fix? From a quick look I doubt using the correct resp in exceptionCaught may solve the issue (sorry if I'm wrong).

        For tests, I think we should try harder with the existing methods. For example, would MiniDFSCluster#shutdownDataNodes right before append enough to fail for the test? GenericTestUtils#assertExceptionContains could also be helpful for the test case.

        Show
        xiaochen Xiao Chen added a comment - Thanks Yuanbo Liu for reporting the issue and providing a fix. Could you explain a little more about your fix? From a quick look I doubt using the correct resp in exceptionCaught may solve the issue (sorry if I'm wrong). For tests, I think we should try harder with the existing methods. For example, would MiniDFSCluster#shutdownDataNodes right before append enough to fail for the test? GenericTestUtils#assertExceptionContains could also be helpful for the test case.
        Hide
        yuanbo Yuanbo Liu added a comment -

        Xiao Chen Thanks for your response.

        From a quick look I doubt using the correct......

        I'm afraid not. The pipeline of transform data is:

        webhdfs-client  ->(1)  webhdfs server in datanode ->(2) hdfs block.
        

        And exceptionCaught only takes effect when pipeline(1) encounter exceptions. The exception of this issue happens in pipeline(2).
        What's more, pipeline(2) relies on DataStreamer, it's a asynchronous data transfer with buffer in another thread. Only when closing OutputStream and flushing data , we have the chance to catch the exception.
        It's not elegant to ignore the exception in my code change by the way, I'll bring up a new way to fix it.

        For example, would MiniDFSCluster#shutdownDataNodes right..

        Good suggestion, It will certainly work in this test case. I will apply it in my next patch.

        Show
        yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for your response. From a quick look I doubt using the correct...... I'm afraid not. The pipeline of transform data is: webhdfs-client ->(1) webhdfs server in datanode ->(2) hdfs block. And exceptionCaught only takes effect when pipeline(1) encounter exceptions. The exception of this issue happens in pipeline(2). What's more, pipeline(2) relies on DataStreamer , it's a asynchronous data transfer with buffer in another thread. Only when closing OutputStream and flushing data , we have the chance to catch the exception. It's not elegant to ignore the exception in my code change by the way, I'll bring up a new way to fix it. For example, would MiniDFSCluster#shutdownDataNodes right.. Good suggestion, It will certainly work in this test case. I will apply it in my next patch.
        Hide
        yuanbo Yuanbo Liu added a comment -

        upload v2 patch for this issue.

        Show
        yuanbo Yuanbo Liu added a comment - upload v2 patch for this issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 45s trunk passed
        +1 compile 0m 53s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 0m 46s trunk passed
        +1 mvninstall 0m 58s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        +1 checkstyle 0m 31s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 54s the patch passed
        +1 javadoc 0m 39s the patch passed
        -1 unit 88m 0s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        109m 52s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones
          hadoop.hdfs.TestSecureEncryptionZoneWithKMS
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
          hadoop.hdfs.TestErasureCodeBenchmarkThroughput
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11195
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842777/HDFS-11195.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 728b7e8b1cf1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4c38f11
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17835/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17835/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 45s trunk passed +1 compile 0m 53s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed +1 checkstyle 0m 31s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 88m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 109m 52s Reason Tests Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones   hadoop.hdfs.TestSecureEncryptionZoneWithKMS   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.TestErasureCodeBenchmarkThroughput Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842777/HDFS-11195.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 728b7e8b1cf1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17835/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17835/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17835/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yuanbo Yuanbo Liu added a comment -

        The test failures are not related, some of them have been addressed in HADOOP-13890.
        Xiao Chen / Wei-Chiu Chuang Can you help me review my latest patch? Thanks in advance.

        Show
        yuanbo Yuanbo Liu added a comment - The test failures are not related, some of them have been addressed in HADOOP-13890 . Xiao Chen / Wei-Chiu Chuang Can you help me review my latest patch? Thanks in advance.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks Yuanbo Liu for reporting this and providing the patch. Now I have one comment:

        HdfsWriter.java
          private void releaseDfsResourcesAndThrow() throws Exception {
            cleanupAndThrow(LOG, out);
            cleanupAndThrow(LOG, client);
          }
        

        If an exception is thrown when closing out, client won't be closed. I'd like to re-use exceptionCaught to close them as follows:

        HdfsWriter#channelRead0
              } catch (Exception cause) {
                exceptionCaught(ctx, cause);
              }
        
        Show
        ajisakaa Akira Ajisaka added a comment - Thanks Yuanbo Liu for reporting this and providing the patch. Now I have one comment: HdfsWriter.java private void releaseDfsResourcesAndThrow() throws Exception { cleanupAndThrow(LOG, out); cleanupAndThrow(LOG, client); } If an exception is thrown when closing out , client won't be closed. I'd like to re-use exceptionCaught to close them as follows: HdfsWriter#channelRead0 } catch (Exception cause) { exceptionCaught(ctx, cause); }
        Hide
        yuanbo Yuanbo Liu added a comment -

        Akira Ajisaka Thanks a lot for your comments !
        Upload v3 patch for this issue.

        Show
        yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks a lot for your comments ! Upload v3 patch for this issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 12m 53s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 0m 38s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 37s the patch passed
        +1 unit 61m 56s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        86m 26s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11195
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843816/HDFS-11195.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c4858aa493c0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ef2dd7b
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17885/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17885/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 53s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 37s the patch passed +1 unit 61m 56s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 86m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843816/HDFS-11195.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4858aa493c0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17885/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17885/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Yuanbo Liu for the explanation and new rev, and Akira Ajisaka for review.

        I think we should also add a log message in exceptionCaught, to make debugging easier. +1 after that.

        Show
        xiaochen Xiao Chen added a comment - Thanks Yuanbo Liu for the explanation and new rev, and Akira Ajisaka for review. I think we should also add a log message in exceptionCaught , to make debugging easier. +1 after that.
        Hide
        yuanbo Yuanbo Liu added a comment -

        Xiao Chen Thanks for your response.
        Upload v4 patch to address your comments.

        Show
        yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for your response. Upload v4 patch to address your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 30s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 49s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 26s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 62m 11s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        87m 13s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11195
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843999/HDFS-11195.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 62c1f446658f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3583413
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17911/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17911/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17911/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 30s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 62m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 87m 13s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843999/HDFS-11195.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 62c1f446658f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3583413 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17911/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17911/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17911/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yuanbo Yuanbo Liu added a comment -

        The test failure is not related to my code change.

        Show
        yuanbo Yuanbo Liu added a comment - The test failure is not related to my code change.
        Hide
        xiaochen Xiao Chen added a comment -

        +1 on patch 4, committing this.

        Show
        xiaochen Xiao Chen added a comment - +1 on patch 4, committing this.
        Hide
        xiaochen Xiao Chen added a comment -

        Failed test look unrelated and passed locally. Committed to trunk, branch-2 and branch-2.8.

        Thanks Yuanbo Liu for the contribution and Akira Ajisaka for the review!

        Show
        xiaochen Xiao Chen added a comment - Failed test look unrelated and passed locally. Committed to trunk, branch-2 and branch-2.8. Thanks Yuanbo Liu for the contribution and Akira Ajisaka for the review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11019 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11019/)
        HDFS-11195. Return error when appending files by webhdfs rest api fails. (xiao: rev 5b7acdd206f5a7d1b7af29b68adaa7587d7d8c43)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/HdfsWriter.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11019 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11019/ ) HDFS-11195 . Return error when appending files by webhdfs rest api fails. (xiao: rev 5b7acdd206f5a7d1b7af29b68adaa7587d7d8c43) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/HdfsWriter.java
        Hide
        yuanbo Yuanbo Liu added a comment - - edited

        Xiao Chen / Akira Ajisaka Thanks a lot!

        Show
        yuanbo Yuanbo Liu added a comment - - edited Xiao Chen / Akira Ajisaka Thanks a lot!

          People

          • Assignee:
            yuanbo Yuanbo Liu
            Reporter:
            yuanbo Yuanbo Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development