Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3577

WebHdfsFileSystem can not read files larger than 24KB

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If reading a file large enough for which the httpserver running webhdfs/httpfs uses chunked transfer encoding (more than 24K in the case of webhdfs), then the WebHdfsFileSystem client fails with an IOException with message Content-Length header is missing.

      It looks like WebHdfsFileSystem is delegating opening of the inputstream to ByteRangeInputStream.URLOpener class, which checks for the Content-Length header, but when using chunked transfer encoding the Content-Length header is not present and the URLOpener.openInputStream() method thrown an exception.

      1. h3577_20120717.patch
        7 kB
        Tsz Wo Nicholas Sze
      2. h3577_20120716.patch
        9 kB
        Tsz Wo Nicholas Sze
      3. h3577_20120714.patch
        9 kB
        Tsz Wo Nicholas Sze
      4. h3577_20120708.patch
        4 kB
        Tsz Wo Nicholas Sze
      5. h3577_20120705.patch
        4 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Alejandro Abdelnur created issue -
          Hide
          Eli Collins added a comment -

          Is this true on branch-1 as well?

          Show
          Eli Collins added a comment - Is this true on branch-1 as well?
          Hide
          Alejandro Abdelnur added a comment -

          @Eli, it doesn't look like, doing a quick check to the ByteRangeInputStream.getInputStream() method the impl is different, no exception begin thrown:

                final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH);
                filelength = (cl == null) ? -1 : Long.parseLong(cl);
                in = connection.getInputStream();
          

          The ByteRangeInputClass seems that had some significant rewriting in Hadoop2.

          Show
          Alejandro Abdelnur added a comment - @Eli, it doesn't look like, doing a quick check to the ByteRangeInputStream.getInputStream() method the impl is different, no exception begin thrown: final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH); filelength = (cl == null ) ? -1 : Long .parseLong(cl); in = connection.getInputStream(); The ByteRangeInputClass seems that had some significant rewriting in Hadoop2.
          Hide
          Eli Collins added a comment -

          I verfied this is an issue on branch-1 as well:

          2012-07-03 18:43:23,150 INFO org.apache.hadoop.tools.DistCp: FAIL CentOS-5.8-x86_64-netinstall.iso : java.io.IOException: Content-Length header is missing
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:125)
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158)
          	at java.io.DataInputStream.read(DataInputStream.java:83)
          	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424)
          	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547)
          	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314)
          	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
          	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
          	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
          	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
          	at org.apache.hadoop.mapred.Child.main(Child.java:260)
          
          Show
          Eli Collins added a comment - I verfied this is an issue on branch-1 as well: 2012-07-03 18:43:23,150 INFO org.apache.hadoop.tools.DistCp: FAIL CentOS-5.8-x86_64-netinstall.iso : java.io.IOException: Content-Length header is missing at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:125) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:424) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547) at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178) at org.apache.hadoop.mapred.Child.main(Child.java:260)
          Eli Collins made changes -
          Field Original Value New Value
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Target Version/s 1.1.0, 2.0.1-alpha [ 12317959, 12321440 ]
          Hide
          Eli Collins added a comment -

          Marking as a blocker since this prevents any reasonable distcp using webhdfs from working.

          Show
          Eli Collins added a comment - Marking as a blocker since this prevents any reasonable distcp using webhdfs from working.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Two potential fixes:

          1. change WebHdfsFileSystem to support chunked transfer encoding;
          2. change server to set content length.

          Which one is better?

          Show
          Tsz Wo Nicholas Sze added a comment - Two potential fixes: change WebHdfsFileSystem to support chunked transfer encoding; change server to set content length. Which one is better?
          Hide
          Alejandro Abdelnur added a comment -

          AFAIK you don't want to do #2 as that triggers in-memory client full buffering of the response. Note that to do #1 you don't have to do much, in Hadoop 2 just don't throw an exception if content length is not set.

          Show
          Alejandro Abdelnur added a comment - AFAIK you don't want to do #2 as that triggers in-memory client full buffering of the response. Note that to do #1 you don't have to do much, in Hadoop 2 just don't throw an exception if content length is not set.
          Tsz Wo Nicholas Sze made changes -
          Release Note Alejandro, thanks for the hint. I will try #1 then.
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Alejandro, thanks for the hint. I will try #1 then.

          Show
          Tsz Wo Nicholas Sze added a comment - Alejandro, thanks for the hint. I will try #1 then.
          Tsz Wo Nicholas Sze made changes -
          Release Note Alejandro, thanks for the hint. I will try #1 then.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h3577_20120705.patch: do not throw exceptions when Content-Length is missing.

          Show
          Tsz Wo Nicholas Sze added a comment - h3577_20120705.patch: do not throw exceptions when Content-Length is missing.
          Tsz Wo Nicholas Sze made changes -
          Attachment h3577_20120705.patch [ 12535316 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535316/h3577_20120705.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2746//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2746//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535316/h3577_20120705.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2746//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2746//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535316/h3577_20120705.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.TestHDFSTrash

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2747//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2747//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535316/h3577_20120705.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2747//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2747//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The file size was 1MB in the test but the block size was only 1kB. Therefore, it created a lot of local files and failed with "java.net.SocketException: Too many open files".

          Show
          Tsz Wo Nicholas Sze added a comment - The file size was 1MB in the test but the block size was only 1kB. Therefore, it created a lot of local files and failed with "java.net.SocketException: Too many open files".
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h3577_20120708.patch: use a larger block size.

          Show
          Tsz Wo Nicholas Sze added a comment - h3577_20120708.patch: use a larger block size.
          Tsz Wo Nicholas Sze made changes -
          Attachment h3577_20120708.patch [ 12535614 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s hdfs client [ 12312928 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535614/h3577_20120708.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.TestDFSClientRetries

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535614/h3577_20120708.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.TestDFSClientRetries +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2761//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          No, no, no! This is reverting a fix for > 32-bit file transfers. I think the correct fix is to require content-length unless chunked encoding is being used.

          Show
          Daryn Sharp added a comment - No, no, no! This is reverting a fix for > 32-bit file transfers. I think the correct fix is to require content-length unless chunked encoding is being used.
          Hide
          Daryn Sharp added a comment -

          Sorry, I freaked out before studying the whole patch. I still think a chunked encoding check should be present unless I'm misunderstanding something. There's also not much use in instantiating a BoundedInputStream w/o a limit.

          Show
          Daryn Sharp added a comment - Sorry, I freaked out before studying the whole patch. I still think a chunked encoding check should be present unless I'm misunderstanding something. There's also not much use in instantiating a BoundedInputStream w/o a limit.
          Hide
          Daryn Sharp added a comment -

          The file size was 1MB in the test but the block size was only 1kB. Therefore, it created a lot of local files and failed with "java.net.SocketException: Too many open files".

          Does this mean there's a fd leak? Or at least a leak during the create request? If so, is the test at fault?

          Show
          Daryn Sharp added a comment - The file size was 1MB in the test but the block size was only 1kB. Therefore, it created a lot of local files and failed with "java.net.SocketException: Too many open files". Does this mean there's a fd leak? Or at least a leak during the create request? If so, is the test at fault?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It turns that the size of each chunk is very small (tens of bytes) when it uses chunked transfer encoding. The client keeps opening new socket for each chunk and then it leads to SocketException. The problem disappear if StreamingOutput is replaced with MessageBodyWriter since it can specify the response size.

          Show
          Tsz Wo Nicholas Sze added a comment - It turns that the size of each chunk is very small (tens of bytes) when it uses chunked transfer encoding. The client keeps opening new socket for each chunk and then it leads to SocketException. The problem disappear if StreamingOutput is replaced with MessageBodyWriter since it can specify the response size.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h3577_20120714.patch: uses MessageBodyWriter in DatanodeWebHdfsMethods.

          Show
          Tsz Wo Nicholas Sze added a comment - h3577_20120714.patch: uses MessageBodyWriter in DatanodeWebHdfsMethods.
          Tsz Wo Nicholas Sze made changes -
          Attachment h3577_20120714.patch [ 12536511 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536511/h3577_20120714.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2824//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2824//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536511/h3577_20120714.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2824//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2824//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          How about a test that copies > 1 block of data so we can detect regressions?

          Show
          Eli Collins added a comment - How about a test that copies > 1 block of data so we can detect regressions?
          Eli Collins made changes -
          Summary webHdfsFileSystem fails to read files with chunked transfer encoding WebHdfsFileSystem can not read files larger than 24KB
          Hide
          Eli Collins added a comment -

          Tucu, do we have an HttpFs test that checks that it can read files big enough to require chunked transfer encoding?

          Show
          Eli Collins added a comment - Tucu, do we have an HttpFs test that checks that it can read files big enough to require chunked transfer encoding?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The failed TestBlocksWithNotEnoughRacks is not related to this.

          > How about a test that copies > 1 block of data so we can detect regressions?

          Hi Eli, thanks for taking a look. The modified test in the patch already has writing and reading > 1 blocks of data. It will fail if the fix is not applied.

          Show
          Tsz Wo Nicholas Sze added a comment - The failed TestBlocksWithNotEnoughRacks is not related to this. > How about a test that copies > 1 block of data so we can detect regressions? Hi Eli, thanks for taking a look. The modified test in the patch already has writing and reading > 1 blocks of data. It will fail if the fix is not applied.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536511/h3577_20120714.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestBackupNode

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2830//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2830//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536511/h3577_20120714.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2830//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2830//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          The BoundedInputStream is a no-op when the ctor w/o a length is used, so I think this:

          final InputStream is = cl == null? new BoundedInputStream(in)
              : new BoundedInputStream(in, Long.parseLong(cl));

          can be:

          final InputStream is = cl == null? in
              : new BoundedInputStream(in, Long.parseLong(cl));

          A chunk size can be specified for a HttpURLConnection and we should be able to enable keep-alive on the socket (I thought it was the default?) to avoid new connections for every chunk. I don't know anything about MessageBodyWriter et al, so if my suggestion isn't feasible and someone else oks the MessageBodyWriter, I'm fine with it.

          Show
          Daryn Sharp added a comment - The BoundedInputStream is a no-op when the ctor w/o a length is used, so I think this: final InputStream is = cl == null ? new BoundedInputStream(in) : new BoundedInputStream(in, Long .parseLong(cl)); can be: final InputStream is = cl == null ? in : new BoundedInputStream(in, Long .parseLong(cl)); A chunk size can be specified for a HttpURLConnection and we should be able to enable keep-alive on the socket (I thought it was the default?) to avoid new connections for every chunk. I don't know anything about MessageBodyWriter et al, so if my suggestion isn't feasible and someone else oks the MessageBodyWriter , I'm fine with it.
          Hide
          Robert Joseph Evans added a comment -

          This impacts branch-0.23 as well. I really would like to see whatever fix happens go into branch-0.23 as well. I applied the latest patch and it looks to apply fairly cleanly. If it does not apply cleanly when checking in the file fix I will be happy to port it.

          Show
          Robert Joseph Evans added a comment - This impacts branch-0.23 as well. I really would like to see whatever fix happens go into branch-0.23 as well. I applied the latest patch and it looks to apply fairly cleanly. If it does not apply cleanly when checking in the file fix I will be happy to port it.
          Robert Joseph Evans made changes -
          Affects Version/s 0.23.3 [ 12320052 ]
          Target Version/s 1.1.0, 2.1.0-alpha [ 12317959, 12321440 ] 1.1.0, 0.23.3, 2.1.0-alpha [ 12317959, 12320052, 12321440 ]
          Hide
          Alejandro Abdelnur added a comment -

          @Eli, chunked encoding is enabled by default in modern servlet containers, it normally kicks in after a threshold (I believe in Tomcat this threshold is 8K), don't know what is the threshold of jetty.

          Another thing to keep in mind, is that the HttpURLConnection fully caches in memory the outputstream (uploading data) and this seems hardcoded in the JDK. This would pose a limitation in writing files through webhdfs/httpfs. A solution may be to refactor WebHdfsFileSystem to use HttpClient (arghh, we may need to tweak hadoop-auth to work with HttpClient).

          Show
          Alejandro Abdelnur added a comment - @Eli, chunked encoding is enabled by default in modern servlet containers, it normally kicks in after a threshold (I believe in Tomcat this threshold is 8K), don't know what is the threshold of jetty. Another thing to keep in mind, is that the HttpURLConnection fully caches in memory the outputstream (uploading data) and this seems hardcoded in the JDK. This would pose a limitation in writing files through webhdfs/httpfs. A solution may be to refactor WebHdfsFileSystem to use HttpClient (arghh, we may need to tweak hadoop-auth to work with HttpClient).
          Rahul Jain made changes -
          Link This issue relates to HDFS-3318 [ HDFS-3318 ]
          Hide
          Rahul Jain added a comment -

          Added link to HDFS-3318: fix of that issue led to the issue described in HDFS-3577

          Show
          Rahul Jain added a comment - Added link to HDFS-3318 : fix of that issue led to the issue described in HDFS-3577
          Hide
          Daryn Sharp added a comment -

          Another thing to keep in mind, is that the HttpURLConnection fully caches in memory the outputstream

          Isn't that only the case for non-chunked responses since it has to compute the content-length? But to avoid that, I think you can use setFixedLengthStreamingMode(length).

          Show
          Daryn Sharp added a comment - Another thing to keep in mind, is that the HttpURLConnection fully caches in memory the outputstream Isn't that only the case for non-chunked responses since it has to compute the content-length? But to avoid that, I think you can use setFixedLengthStreamingMode(length) .
          Tsz Wo Nicholas Sze made changes -
          Link This issue is duplicated by HDFS-3671 [ HDFS-3671 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h3577_20120716.patch: do not use BoundedInputStream when content-length is not found.

          Show
          Tsz Wo Nicholas Sze added a comment - h3577_20120716.patch: do not use BoundedInputStream when content-length is not found.
          Tsz Wo Nicholas Sze made changes -
          Attachment h3577_20120716.patch [ 12536761 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536761/h3577_20120716.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2838//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2838//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536761/h3577_20120716.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2838//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2838//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          @daryn, http://docs.oracle.com/javase/6/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode(int), works only if you know the length in advance, when doing a FS create() you don't know how many bytes will be written.

          Show
          Alejandro Abdelnur added a comment - @daryn, http://docs.oracle.com/javase/6/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode(int ), works only if you know the length in advance, when doing a FS create() you don't know how many bytes will be written.
          Hide
          Alejandro Abdelnur added a comment -

          It seems that using http://docs.oracle.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int) would do the trick, if the server side (jetty & tomcat) support it.

          Show
          Alejandro Abdelnur added a comment - It seems that using http://docs.oracle.com/javase/6/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode(int ) would do the trick, if the server side (jetty & tomcat) support it.
          Hide
          Daryn Sharp added a comment -

          If the content-length check is removed and content-length is not present, transfers >2GB will fail unless the client's read timeout is greater than the server's connection idle timeout (200s), and partial downloads will be considered successful.

          I'd suggest the check be left in place on this jira, but it only generates an exception if it's not a chunked download. We can then decide what to do on HDFS-3671.

          Show
          Daryn Sharp added a comment - If the content-length check is removed and content-length is not present, transfers >2GB will fail unless the client's read timeout is greater than the server's connection idle timeout (200s), and partial downloads will be considered successful. I'd suggest the check be left in place on this jira, but it only generates an exception if it's not a chunked download. We can then decide what to do on HDFS-3671 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Okay, let's revert the changes in ByteRangeInputStream. Here is a new patch.

          h3577_20120717.patch

          Show
          Tsz Wo Nicholas Sze added a comment - Okay, let's revert the changes in ByteRangeInputStream. Here is a new patch. h3577_20120717.patch
          Tsz Wo Nicholas Sze made changes -
          Attachment h3577_20120717.patch [ 12536946 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is broken by HDFS-3318 [ HDFS-3318 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3318 [ HDFS-3318 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3671 [ HDFS-3671 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is duplicated by HDFS-3671 [ HDFS-3671 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536946/h3577_20120717.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2851//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2851//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536946/h3577_20120717.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2851//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2851//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 Looks good to me! Might want to alter the description to indicate it's fixing the problem of making new connections for every chunk and exhausting fds.

          Show
          Daryn Sharp added a comment - +1 Looks good to me! Might want to alter the description to indicate it's fixing the problem of making new connections for every chunk and exhausting fds.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2495 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2495/)
          HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2495 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2495/ ) HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2560 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2560/)
          HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2560 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2560/ ) HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Daryn for the review.

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Daryn for the review. I have committed this.
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.23.3 [ 12320052 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2517/)
          HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2517/ ) HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Eli Collins added a comment -

          Thanks Nicholas. This needs to be fixed in branch-1 as well, mind posting a patch? IMO this is a 1.1 blocker as w/o it we've broken compat with earlier 20 releases and you can't distcp with webdhfs.

          Show
          Eli Collins added a comment - Thanks Nicholas. This needs to be fixed in branch-1 as well, mind posting a patch? IMO this is a 1.1 blocker as w/o it we've broken compat with earlier 20 releases and you can't distcp with webdhfs.
          Eli Collins made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Eli, the Content-Length exception was introduced by HDFS-3318 which was committed to 0.23 but not branch-1. So branch-1 does not have the problem on transferring files > 24kB. However, it may have problem on transferring files > 2GB since it does not have HDFS-3318. If it is the case, we should backport both HDFS-3318 and this.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Eli, the Content-Length exception was introduced by HDFS-3318 which was committed to 0.23 but not branch-1. So branch-1 does not have the problem on transferring files > 24kB. However, it may have problem on transferring files > 2GB since it does not have HDFS-3318 . If it is the case, we should backport both HDFS-3318 and this.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I will test > 2GB files in branch-1.

          Show
          Tsz Wo Nicholas Sze added a comment - I will test > 2GB files in branch-1.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1108/)
          HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1108/ ) HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #318 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/318/)
          svn merge -c 1362976 from trunk for HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362981)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362981
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #318 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/318/ ) svn merge -c 1362976 from trunk for HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362981) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362981 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1141/)
          HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1141/ ) HDFS-3577 . In DatanodeWebHdfsMethods, use MessageBodyWriter instead of StreamingOutput, otherwise, it will fail to transfer large files. (Revision 1362976) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362976 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/OpenEntity.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
          Hide
          Eli Collins added a comment -

          Thanks Nicholas, I forgot 3318 was not in branch-1. I pinged HDFS-3671 since we're not removing the Content-Length check here.

          Show
          Eli Collins added a comment - Thanks Nicholas, I forgot 3318 was not in branch-1. I pinged HDFS-3671 since we're not removing the Content-Length check here.
          Hide
          Eli Collins added a comment -

          I merged this to branch-2.1.0-alpha.

          Show
          Eli Collins added a comment - I merged this to branch-2.1.0-alpha.
          Eli Collins made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Target Version/s 1.1.0, 0.23.3, 2.1.0-alpha [ 12317959, 12320052, 12321440 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Resolution Fixed [ 1 ]
          Hide
          Eli Collins added a comment -

          Hey Nicholas,

          Did you test this with distcp? Trying to distcp from a recent trunk build with this change still fails with Content-Length header is missing. Hadoop fs -get using webhdfs with the same file works.

          12/07/19 23:56:43 INFO mapreduce.Job: Task Id : attempt_1342766959778_0002_m_000000_0, Status : FAILED
          Error: java.io.IOException: File copy failed: webhdfs://eli-thinkpad:50070/user/eli/data1/big.iso --> hdfs://localhost:8020/user/eli/data4/data1/big.iso
          	at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
          	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
          	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
          	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
          	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
          	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
          	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
          	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149)
          Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://eli-thinkpad:50070/user/eli/data1/big.iso to hdfs://localhost:8020/user/eli/data4/data1/big.iso
          	at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
          	at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
          	... 10 more
          Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: Content-Length header is missing
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:201)
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:167)
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:112)
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:90)
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:71)
          	at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
          	... 11 more
          Caused by: java.io.IOException: Content-Length header is missing
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:125)
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
          	at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158)
          	at java.io.DataInputStream.read(DataInputStream.java:132)
          	at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
          	at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
          	at java.io.FilterInputStream.read(FilterInputStream.java:90)
          	at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:70)
          	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:198)
          	... 16 more
          
          Show
          Eli Collins added a comment - Hey Nicholas, Did you test this with distcp? Trying to distcp from a recent trunk build with this change still fails with Content-Length header is missing . Hadoop fs -get using webhdfs with the same file works. 12/07/19 23:56:43 INFO mapreduce.Job: Task Id : attempt_1342766959778_0002_m_000000_0, Status : FAILED Error: java.io.IOException: File copy failed: webhdfs://eli-thinkpad:50070/user/eli/data1/big.iso --> hdfs://localhost:8020/user/eli/data4/data1/big.iso at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149) Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://eli-thinkpad:50070/user/eli/data1/big.iso to hdfs://localhost:8020/user/eli/data4/data1/big.iso at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: java.io.IOException: Content-Length header is missing at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:201) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:167) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:112) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:90) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:71) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.io.IOException: Content-Length header is missing at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:125) at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103) at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158) at java.io.DataInputStream.read(DataInputStream.java:132) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.FilterInputStream.read(FilterInputStream.java:90) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:70) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:198) ... 16 more
          Hide
          Eli Collins added a comment -

          Per offline conversation with Daryn the reason this isn't an issue is that HDFS-3166 (Add timeout to Hftp connections) isn't in branch-1 (w/o the timeout we just get a 200s lag, not a failure) and 3166 exposed HDFS-3318 (Hftp hangs on transfers >2GB) which introduces the check that will fail if the content length header is not set (ie 20.x).

          Show
          Eli Collins added a comment - Per offline conversation with Daryn the reason this isn't an issue is that HDFS-3166 (Add timeout to Hftp connections) isn't in branch-1 (w/o the timeout we just get a 200s lag, not a failure) and 3166 exposed HDFS-3318 (Hftp hangs on transfers >2GB) which introduces the check that will fail if the content length header is not set (ie 20.x).
          Hide
          Daryn Sharp added a comment -

          Yes, HDFS-3166 (add timeouts) exposed the 200s tail on >2GB files caused by a java bug. The content-length has to be known in order to workaround the java bug, and thus avoid the read timeout.

          What I think can be done:

          • If chunked, eliminate the content-length requirement
          • If not chunked, and no content-length, obtain the length from a file stat or a HEAD, etc
          Show
          Daryn Sharp added a comment - Yes, HDFS-3166 (add timeouts) exposed the 200s tail on >2GB files caused by a java bug. The content-length has to be known in order to workaround the java bug, and thus avoid the read timeout. What I think can be done: If chunked, eliminate the content-length requirement If not chunked, and no content-length, obtain the length from a file stat or a HEAD, etc
          Hide
          Eli Collins added a comment -

          Shouldn't be an issue in branch-1 I mean. I verified that on a branch-1 build from today distcp using webhdfs of a directory that contains a 3gb file works. So looks like this is just an issue on trunk/2.x, though per my comment above the same distcp does not work even with 3577 applied.

          Show
          Eli Collins added a comment - Shouldn't be an issue in branch-1 I mean. I verified that on a branch-1 build from today distcp using webhdfs of a directory that contains a 3gb file works. So looks like this is just an issue on trunk/2.x, though per my comment above the same distcp does not work even with 3577 applied.
          Eli Collins made changes -
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ] Eli Collins [ eli ]
          Eli Collins made changes -
          Link This issue relates to HDFS-3788 [ HDFS-3788 ]
          Hide
          Eli Collins added a comment -

          Btw webhdfs is still broken for large files when using distcp. I confirmed on the latest trunk build and filed HDFS-3788.

          Show
          Eli Collins added a comment - Btw webhdfs is still broken for large files when using distcp. I confirmed on the latest trunk build and filed HDFS-3788 .
          Tsz Wo Nicholas Sze made changes -
          Assignee Eli Collins [ eli ] Tsz Wo (Nicholas), SZE [ szetszwo ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Vinayakumar B made changes -
          Component/s webhdfs [ 12319200 ]
          Component/s hdfs-client [ 12312928 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-4715 [ HDFS-4715 ]

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development