Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2606

webhdfs client filesystem impl must set the content-type header for create/append

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.1
    • Fix Version/s: 0.23.1
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the content-type header is not being set and for some reason for append it is being set to the form encoded content type making jersey parameter parsing fail.

      For this and to avoid any kind of proxy transcoding the content-type should be set to binary.

        conn.setRequestProperty("Content-Type", "application/octet-stream");
      
      1. HDFS-2606.patch
        0.8 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Alejandro Abdelnur added a comment -

          a testcase, using webhdfs filesystem is failing on append because of this issue.

          Show
          Alejandro Abdelnur added a comment - a testcase, using webhdfs filesystem is failing on append because of this issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12505560/HDFS-2606.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1614//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505560/HDFS-2606.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1614//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1416 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1416/)
          HDFS-2606. webhdfs client filesystem impl must set the content-type header for create/append. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1416 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1416/ ) HDFS-2606 . webhdfs client filesystem impl must set the content-type header for create/append. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #230 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/230/)
          Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #230 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/230/ ) Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #232 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/232/)
          Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #232 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/232/ ) Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1343 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1343/)
          HDFS-2606. webhdfs client filesystem impl must set the content-type header for create/append. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1343 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1343/ ) HDFS-2606 . webhdfs client filesystem impl must set the content-type header for create/append. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #241 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/241/)
          Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #241 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/241/ ) Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1366/)
          HDFS-2606. webhdfs client filesystem impl must set the content-type header for create/append. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1366/ ) HDFS-2606 . webhdfs client filesystem impl must set the content-type header for create/append. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #93 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/93/)
          Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #93 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/93/ ) Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #880 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/880/)
          HDFS-2606. webhdfs client filesystem impl must set the content-type header for create/append. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #880 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/880/ ) HDFS-2606 . webhdfs client filesystem impl must set the content-type header for create/append. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/109/)
          Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/109/ ) Merge -r 1208156:1208157 from trunk to branch. FIXES: HDFS-2606 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208158 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #913 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/913/)
          HDFS-2606. webhdfs client filesystem impl must set the content-type header for create/append. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #913 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/913/ ) HDFS-2606 . webhdfs client filesystem impl must set the content-type header for create/append. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208157 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          Alejandro Abdelnur added a comment -

          Nicholas, thanks to help me troubleshoot and identify the problem.

          Show
          Alejandro Abdelnur added a comment - Nicholas, thanks to help me troubleshoot and identify the problem.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Alejandro, you are welcome!

          BTW, I suggest we also change the server side so that append will work if the content-type is not specified. This follows the Postel Law as you mentioned previously.

          Show
          Tsz Wo Nicholas Sze added a comment - Alejandro, you are welcome! BTW, I suggest we also change the server side so that append will work if the content-type is not specified. This follows the Postel Law as you mentioned previously.
          Hide
          Alejandro Abdelnur added a comment -

          The thing is that the Jersey is processing eagerly that content-type as form input.

          The correct thing to do is that the server should reject a request that does not have a application/octet-stream content type. This ensure the data is not modified in transit by proxies. Other content-types (such as text based ones) don't guarantee this.

          Show
          Alejandro Abdelnur added a comment - The thing is that the Jersey is processing eagerly that content-type as form input. The correct thing to do is that the server should reject a request that does not have a application/octet-stream content type. This ensure the data is not modified in transit by proxies. Other content-types (such as text based ones) don't guarantee this.

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development