Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3794

WebHDFS Open used with Offset returns the original (and incorrect) Content Length in the HTTP Header.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, 1-win, 1.3.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: webhdfs
    • Labels:
      None

      Description

      When an offset is specified, the HTTP header Content Length still contains the original file size. e.g. if the original file is 100 bytes, and the offset specified it 10, then HTTP Content Length ought to be 90. Currently it is still returned as 100.
      This causes curl to give error 18, and JAVA to throw ConnectionClosedException

      1. HDFS-3794.patch
        1.0 kB
        Ravi Prakash
      2. HDFS-3794.patch
        0.9 kB
        Ravi Prakash
      3. HDFS-3794_branch-1.patch
        0.7 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        Chris Nauroth added a comment -

        I committed this to branch-1 and branch-1-win. Thanks, Nicholas and Jing!

        Show
        Chris Nauroth added a comment - I committed this to branch-1 and branch-1-win. Thanks, Nicholas and Jing!
        Hide
        Chris Nauroth added a comment -

        +1 for the branch-1 patch.

        I'm also going to run some tests on Windows with this patch applied to branch-1-win. Assuming that goes well, I'll commit to branch-1 and branch-1-win soon.

        Show
        Chris Nauroth added a comment - +1 for the branch-1 patch. I'm also going to run some tests on Windows with this patch applied to branch-1-win. Assuming that goes well, I'll commit to branch-1 and branch-1-win soon.
        Hide
        Jing Zhao added a comment -

        +1 for the branch-1 patch.

        Show
        Jing Zhao added a comment - +1 for the branch-1 patch.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        HDFS-3794_branch-1.patch: for branch-1.

        Show
        Tsz Wo Nicholas Sze added a comment - HDFS-3794 _branch-1.patch: for branch-1.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #345 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/345/)
        svn merge -c 1373193 FIXES: HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373399)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373399
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #345 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/345/ ) svn merge -c 1373193 FIXES: HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373399) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373399 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Daryn Sharp added a comment -

        Merged to branch 23.

        Show
        Daryn Sharp added a comment - Merged to branch 23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/)
        HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/ ) HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/)
        HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/ ) HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2643/)
        HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2643/ ) HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2578 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2578/)
        HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2578 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2578/ ) HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2604 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2604/)
        HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2604 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2604/ ) HDFS-3794 . WebHDFS OPEN returns the incorrect Content-Length in the HTTP header when offset is specified and length is omitted. Contributed by Ravi Prakash (Revision 1373193) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373193 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Ravi!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Ravi!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 The new patch looks good.

        The failed test is not related.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 The new patch looks good. The failed test is not related.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12540891/HDFS-3794.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3007//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3007//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540891/HDFS-3794.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3007//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3007//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        I tested the modified patch and it too worked in all three previous cases.

        Show
        Ravi Prakash added a comment - I tested the modified patch and it too worked in all three previous cases.
        Hide
        Ravi Prakash added a comment -

        Thanks a lot Nicholas! I'm afraid I don't know enough about the code. I'll defer to you on this! I'm attaching the modified patch with the change you suggested.

        Thanks Daryn. It discovers an out of range offset and throws an exception before reaching this method.

        $ curl -L "http://HOST:PORT/webhdfs/v1/somePath/someFile?op=OPEN&offset=457236547" 
        {"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Offset=457236547 out of the range [0, 457236477); OPEN, path=/somePath/someFile"}}
        
        Show
        Ravi Prakash added a comment - Thanks a lot Nicholas! I'm afraid I don't know enough about the code. I'll defer to you on this! I'm attaching the modified patch with the change you suggested. Thanks Daryn. It discovers an out of range offset and throws an exception before reaching this method. $ curl -L "http://HOST:PORT/webhdfs/v1/somePath/someFile?op=OPEN&offset=457236547" {"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Offset=457236547 out of the range [0, 457236477); OPEN, path=/somePath/someFile"}}
        Hide
        Daryn Sharp added a comment -

        Shouldn't it check if length - offset goes negative? Or is that checked elsewhere?

        Show
        Daryn Sharp added a comment - Shouldn't it check if length - offset goes negative? Or is that checked elsewhere?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Took a second look at the patch. The length HTTP query parameter means "The number of bytes to be processed" according to the API. So it does not need to minus offset and the code should be

              final long n = length.getValue() != null? length.getValue()
                : in.getVisibleLength() - offset.getValue();
        

        What do you think?

        Show
        Tsz Wo Nicholas Sze added a comment - Took a second look at the patch. The length HTTP query parameter means "The number of bytes to be processed" according to the API . So it does not need to minus offset and the code should be final long n = length.getValue() != null ? length.getValue() : in.getVisibleLength() - offset.getValue(); What do you think?
        Hide
        Ravi Prakash added a comment -

        Thanks a lot Nicholas! Could you please commit the patch?

        Credit goes to Phil Su for filing the issue!

        Show
        Ravi Prakash added a comment - Thanks a lot Nicholas! Could you please commit the patch? Credit goes to Phil Su for filing the issue!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The findbugs warnings and the failed tests are not related.

        Show
        Tsz Wo Nicholas Sze added a comment - The findbugs warnings and the failed tests are not related.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12540778/HDFS-3794.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540778/HDFS-3794.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2997//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 on the patch. Good catch!

        Show
        Tsz Wo Nicholas Sze added a comment - +1 on the patch. Good catch!
        Hide
        Ravi Prakash added a comment -

        The same patch applies to branch 0.23, 2, and trunk.

        Show
        Ravi Prakash added a comment - The same patch applies to branch 0.23, 2, and trunk.
        Hide
        Ravi Prakash added a comment -

        Attaching a patch that fixes the issue. Its too trivial to write a unit test (which will have to be pretty complicated :'( ... I tried briefly)
        Here's the testing I did
        1. Small file with offset. Worked
        2. Big file (multiple blocks) with offset. Worked
        3. Big file with offset greater than file size. Correctly threw a RemoteException

        Show
        Ravi Prakash added a comment - Attaching a patch that fixes the issue. Its too trivial to write a unit test (which will have to be pretty complicated :'( ... I tried briefly) Here's the testing I did 1. Small file with offset. Worked 2. Big file (multiple blocks) with offset. Worked 3. Big file with offset greater than file size. Correctly threw a RemoteException
        Hide
        Ravi Prakash added a comment -
        e.g. $ curl -L "http://HOST:PORT/webhdfs/v1/somePath/someFile?op=OPEN&offset=10" 
        curl: (18) transfer closed with 10 bytes remaining to read
        
        Show
        Ravi Prakash added a comment - e.g. $ curl -L "http://HOST:PORT/webhdfs/v1/somePath/someFile?op=OPEN&offset=10" curl: (18) transfer closed with 10 bytes remaining to read

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development