Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4715

Backport HDFS-3577 and other related WebHDFS issue to branch-1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them can fix some WebHDFS performance issues in branch-1.

      1. HDFS-4751.1.patch
        15 kB
        Mark Wagner
      2. HDFS-4751.2.patch
        15 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Tsz Wo Nicholas Sze created issue -
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Summary Backport HDFS-3577 and other related WebHDFS to branch-1 Backport HDFS-3577 and other related WebHDFS issue to branch-1
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-4549 [ HDFS-4549 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3318 [ HDFS-3318 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3577 [ HDFS-3577 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3788 [ HDFS-3788 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch (HDFS-4549.2.patch) looks different from the current trunk code. For backporting, we would try to backport the code exactly so that it is easier to maintain the code. Could you make the backport patch the same as the original patches? Some other comments:

          • The import org.apache.log4j.helpers.BoundedFIFO is not used.
          • the parameter in update(..) should be the read length but not the read value.
            +    return update(getInputStream().read())
            
          • the filelength in udpate(..) may be null
          Show
          Tsz Wo Nicholas Sze added a comment - The patch ( HDFS-4549 .2.patch) looks different from the current trunk code. For backporting, we would try to backport the code exactly so that it is easier to maintain the code. Could you make the backport patch the same as the original patches? Some other comments: The import org.apache.log4j.helpers.BoundedFIFO is not used. the parameter in update(..) should be the read length but not the read value. + return update(getInputStream().read()) the filelength in udpate(..) may be null
          Hide
          Mark Wagner added a comment -

          Addressed your points, and got things as close to trunk as possible.

          Show
          Mark Wagner added a comment - Addressed your points, and got things as close to trunk as possible.
          Mark Wagner made changes -
          Attachment HDFS-4751.1.patch [ 12579452 ]
          Mark Wagner made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579452/HDFS-4751.1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4278//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579452/HDFS-4751.1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4278//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Mark, patch looks good. +1

          Two minor problems:

          • There is an additional System.out.println("length of response: " + e.length) in OpenEntity.
          • In ByteRangeInputStream, the update(..) header splits into two lines. It should be
            +  private int update(final int n) throws IOException {
            -      throws IOException {
            

            instead of

             +  private int update(final int n)
                    throws IOException {
            

          Since both are very minor and I have changed them in my machine, let me post a patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Mark, patch looks good. +1 Two minor problems: There is an additional System.out.println("length of response: " + e.length) in OpenEntity. In ByteRangeInputStream, the update(..) header splits into two lines. It should be + private int update( final int n) throws IOException { - throws IOException { instead of + private int update( final int n) throws IOException { Since both are very minor and I have changed them in my machine, let me post a patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS-4751.2.patch: addresses above two minor problems.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS-4751 .2.patch: addresses above two minor problems.
          Tsz Wo Nicholas Sze made changes -
          Attachment HDFS-4751.2.patch [ 12579855 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Mark, since Jenkins won't be able to test the patch, could you run "ant test-patch" and "ant test-core" and then post the results?

          Show
          Tsz Wo Nicholas Sze added a comment - Mark, since Jenkins won't be able to test the patch, could you run "ant test-patch" and "ant test-core" and then post the results?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579855/HDFS-4751.2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4283//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579855/HDFS-4751.2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4283//console This message is automatically generated.
          Tsz Wo Nicholas Sze made changes -
          Attachment HDFS-4751.2.patch [ 12579855 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment HDFS-4751.2.patch [ 12579858 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579858/HDFS-4751.2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4284//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579858/HDFS-4751.2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4284//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          test-patch results:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 8 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings.
          

          The findbugs warnings were not introduced by the patch.

          Show
          Tsz Wo Nicholas Sze added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 17 new Findbugs (version 1.3.9) warnings. The findbugs warnings were not introduced by the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          All unit tests passed in my machine except TestCLI. It is not related; see HADOOP-9492

          Show
          Tsz Wo Nicholas Sze added a comment - All unit tests passed in my machine except TestCLI. It is not related; see HADOOP-9492
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Mark!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Mark!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Resolution Fixed [ 1 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1h 54m 1 Mark Wagner 19/Apr/13 00:20
          Patch Available Patch Available Resolved Resolved
          3d 19h 40m 1 Tsz Wo Nicholas Sze 22/Apr/13 20:00
          Resolved Resolved Closed Closed
          22d 10h 15m 1 Matt Foley 15/May/13 06:16

            People

            • Assignee:
              Mark Wagner
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development