Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4715

Backport HDFS-3577 and other related WebHDFS issue to branch-1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them can fix some WebHDFS performance issues in branch-1.

      1. HDFS-4751.2.patch
        15 kB
        Tsz Wo Nicholas Sze
      2. HDFS-4751.1.patch
        15 kB
        Mark Wagner

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Mark!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Mark!
          Hide
          Tsz Wo Nicholas Sze added a comment -

          All unit tests passed in my machine except TestCLI. It is not related; see HADOOP-9492

          Show
          Tsz Wo Nicholas Sze added a comment - All unit tests passed in my machine except TestCLI. It is not related; see HADOOP-9492
          Hide
          Tsz Wo Nicholas Sze added a comment -

          test-patch results:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 8 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
          

          The findbugs warnings were not introduced by the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings. The findbugs warnings were not introduced by the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579858/HDFS-4751.2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4284//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579858/HDFS-4751.2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4284//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579855/HDFS-4751.2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4283//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579855/HDFS-4751.2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4283//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Mark, since Jenkins won't be able to test the patch, could you run "ant test-patch" and "ant test-core" and then post the results?

          Show
          Tsz Wo Nicholas Sze added a comment - Mark, since Jenkins won't be able to test the patch, could you run "ant test-patch" and "ant test-core" and then post the results?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS-4751.2.patch: addresses above two minor problems.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS-4751 .2.patch: addresses above two minor problems.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Mark, patch looks good. +1

          Two minor problems:

          • There is an additional System.out.println("length of response: " + e.length) in OpenEntity.
          • In ByteRangeInputStream, the update(..) header splits into two lines. It should be
            +  private int update(final int n) throws IOException {
            -      throws IOException {
            

            instead of

             +  private int update(final int n)
                    throws IOException {
            

          Since both are very minor and I have changed them in my machine, let me post a patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Mark, patch looks good. +1 Two minor problems: There is an additional System.out.println("length of response: " + e.length) in OpenEntity. In ByteRangeInputStream, the update(..) header splits into two lines. It should be + private int update( final int n) throws IOException { - throws IOException { instead of + private int update( final int n) throws IOException { Since both are very minor and I have changed them in my machine, let me post a patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579452/HDFS-4751.1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4278//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579452/HDFS-4751.1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4278//console This message is automatically generated.
          Hide
          Mark Wagner added a comment -

          Addressed your points, and got things as close to trunk as possible.

          Show
          Mark Wagner added a comment - Addressed your points, and got things as close to trunk as possible.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch (HDFS-4549.2.patch) looks different from the current trunk code. For backporting, we would try to backport the code exactly so that it is easier to maintain the code. Could you make the backport patch the same as the original patches? Some other comments:

          • The import org.apache.log4j.helpers.BoundedFIFO is not used.
          • the parameter in update(..) should be the read length but not the read value.
            +    return update(getInputStream().read())
            
          • the filelength in udpate(..) may be null
          Show
          Tsz Wo Nicholas Sze added a comment - The patch ( HDFS-4549 .2.patch) looks different from the current trunk code. For backporting, we would try to backport the code exactly so that it is easier to maintain the code. Could you make the backport patch the same as the original patches? Some other comments: The import org.apache.log4j.helpers.BoundedFIFO is not used. the parameter in update(..) should be the read length but not the read value. + return update(getInputStream().read()) the filelength in udpate(..) may be null

            People

            • Assignee:
              Mark Wagner
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development