Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.3, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Target Version/s:

      Description

      Hftp transfers >2GB hang after the transfer is complete. The problem appears to be caused by java internally using an int for the content length. When it overflows 2GB, it won't check the bounds of the reads on the input stream. The client continues reading after all data is received, and the client blocks until the server times out the connection – many minutes later. In conjunction with hftp timeouts, all transfers >2G fail with a read timeout.

      1. HDFS-3318.patch
        2 kB
        Daryn Sharp
      2. HDFS-3318-1.patch
        3 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          Wrap the url's input stream in a stream bounded with a long based on content length.

          Add impl to pass through the other 2 read methods to the bounded stream. This yields a ~1.4X improvement in transfer times.

          Show
          Daryn Sharp added a comment - Wrap the url's input stream in a stream bounded with a long based on content length. Add impl to pass through the other 2 read methods to the bounded stream. This yields a ~1.4X improvement in transfer times.
          Hide
          Daryn Sharp added a comment -

          Patch does not include tests due to the infeasibility of testing the transfer time/success of a 2GB+ file. It's been manually tested with large customer files that are currently failing to copy.

          Show
          Daryn Sharp added a comment - Patch does not include tests due to the infeasibility of testing the transfer time/success of a 2GB+ file. It's been manually tested with large customer files that are currently failing to copy.
          Hide
          Kihwal Lee added a comment -

          Before introduction of the client-side timeout in hftp, server-side would timeout in 200 seconds, which is the jetty keepalive timeout. Currently when the client-side times out, which is smaller than 200 seconds, hftp client thinks transfer has failed since it does not detect the end of transfer based on the content length header. This doesn't seem to happen when the file size is < 2GB. HttpURLConnection.getContentLength() returns an int (max: 2^32-1) and it might be internally keeping track of progress as long as content-length is < 2GB.

          As a side effect of the fix, it will shed 200 seconds off transfer times for files bigger than 2GB (for pre hftp client timeout), since it will no longer wait for the server side to close the connection.

          Show
          Kihwal Lee added a comment - Before introduction of the client-side timeout in hftp, server-side would timeout in 200 seconds, which is the jetty keepalive timeout. Currently when the client-side times out, which is smaller than 200 seconds, hftp client thinks transfer has failed since it does not detect the end of transfer based on the content length header. This doesn't seem to happen when the file size is < 2GB. HttpURLConnection.getContentLength() returns an int (max: 2^32-1) and it might be internally keeping track of progress as long as content-length is < 2GB. As a side effect of the fix, it will shed 200 seconds off transfer times for files bigger than 2GB (for pre hftp client timeout), since it will no longer wait for the server side to close the connection.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524067/HDFS-3318.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream
          org.apache.hadoop.fs.TestUrlStreamHandler
          org.apache.hadoop.hdfs.TestByteRangeInputStream

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2324//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2324//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524067/HDFS-3318.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.hdfs.TestByteRangeInputStream +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2324//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2324//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Hi Daryn,

          Can streamLength > 2^31 ? Would in then be bounded by that higher number and still cause issues?

          Why are you Overriding

            @Override
            public int read(byte[] b) throws IOException {
              return read(b, 0, b.length);
            }
          

          Wouldn't the original implementation be picked up from InputStream which has exactly the same code? I tested with this short program and it prints exactly what Michael Jackson used to say he is.

          class A {
            public void printA() {
              System.out.println("A");
              printC();
            }
            public void printC() {
              System.out.println("C");
            }
          }
          
          class B extends A {
            @Override
            public void printC() {
              System.out.println("D");
            }
            
            public void printB() {
              System.out.println("B");
              printA();
            }
          }
          
          public class TestJAVA {
            public static void main(String arg[]) {
              B b = new B();
              b.printB();
            }
          }
          
          Show
          Ravi Prakash added a comment - Hi Daryn, Can streamLength > 2^31 ? Would in then be bounded by that higher number and still cause issues? Why are you Overriding @Override public int read(byte[] b) throws IOException { return read(b, 0, b.length); } Wouldn't the original implementation be picked up from InputStream which has exactly the same code? I tested with this short program and it prints exactly what Michael Jackson used to say he is. class A { public void printA() { System.out.println("A"); printC(); } public void printC() { System.out.println("C"); } } class B extends A { @Override public void printC() { System.out.println("D"); } public void printB() { System.out.println("B"); printA(); } } public class TestJAVA { public static void main(String arg[]) { B b = new B(); b.printB(); } }
          Hide
          Daryn Sharp added a comment -

          I doubt the per-read buffer is going to be >2GB for at least 5-10 years. By that time, I think java will have fixed the issue.

          Show
          Daryn Sharp added a comment - I doubt the per-read buffer is going to be >2GB for at least 5-10 years. By that time, I think java will have fixed the issue.
          Hide
          Robert Joseph Evans added a comment -

          I looked at the patch and it looks good to me. I agree with Ravi that we do not need to override read(byte[] b). I am a +1 (non-binding) on this.

          Show
          Robert Joseph Evans added a comment - I looked at the patch and it looks good to me. I agree with Ravi that we do not need to override read(byte[] b). I am a +1 (non-binding) on this.
          Hide
          Daryn Sharp added a comment -

          Removed read(byte[]) and fixed related tests.

          Another jira broke TestUrlStreamHandler. Somehow there are no more fs.$scheme.impl keys in the conf. There are only fs.AbstractFileSystem.$scheme.impl keys.

          Show
          Daryn Sharp added a comment - Removed read(byte[]) and fixed related tests. Another jira broke TestUrlStreamHandler . Somehow there are no more fs.$scheme.impl keys in the conf. There are only fs.AbstractFileSystem.$scheme.impl keys.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524289/HDFS-3318-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.TestUrlStreamHandler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2328//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2328//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524289/HDFS-3318-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.TestUrlStreamHandler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2328//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2328//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Talked offline with Daryn. He has a way of removing any of my doubts Thanks Daryn! +1

          Show
          Ravi Prakash added a comment - Talked offline with Daryn. He has a way of removing any of my doubts Thanks Daryn! +1
          Hide
          Aaron T. Myers added a comment -

          I think the patch largely looks good. I'm confused, however, by the change of how "filelength" is determined.

          It changed from this:

          final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH);
          filelength = (cl == null) ? -1 : Long.parseLong(cl);
          

          To this:

          final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH);
          ...
          final long streamlength = Long.parseLong(cl);
          filelength = startPos + streamlength;
          

          Why does the filelength now begin at startPos? That change seems unrelated to this issue. Or am I missing something?

          +1 once this question is addressed.

          Show
          Aaron T. Myers added a comment - I think the patch largely looks good. I'm confused, however, by the change of how "filelength" is determined. It changed from this: final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH); filelength = (cl == null ) ? -1 : Long .parseLong(cl); To this: final String cl = connection.getHeaderField(StreamFile.CONTENT_LENGTH); ... final long streamlength = Long .parseLong(cl); filelength = startPos + streamlength; Why does the filelength now begin at startPos? That change seems unrelated to this issue. Or am I missing something? +1 once this question is addressed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I think the previous filelength works only if startPos is zero. It is a bug.

          Show
          Tsz Wo Nicholas Sze added a comment - I think the previous filelength works only if startPos is zero. It is a bug.
          Hide
          Daryn Sharp added a comment -

          Why does the filelength now begin at startPos?

          It's another bug related to successfully reading the stream that I didn't fully fix, but fixed "enough". When EOF is encountered, it checks

          if (currentPos < filelength) { EOFException } 

          to decide if there was a premature EOF. currentPos and filelength are not relative to startPos, thus it's not valid to compare the current pos to the stream length.

          Ex. I have 128 bytes. I seek 100 bytes into it. The remaining content-length is 28. My file length is not 28 bytes! I read more 10 bytes and the connection unexpectedly closes. The broken premature EOF condition fails to detect the fault because (110 < 28) is false. The correct check is (110 < 100+28).

                filelength
          ------------------------
                 ^----------------
          startPos  content-length
          

          I can file a separate jira for this 1-line fix if you'd like.

          Show
          Daryn Sharp added a comment - Why does the filelength now begin at startPos? It's another bug related to successfully reading the stream that I didn't fully fix, but fixed "enough". When EOF is encountered, it checks if (currentPos < filelength) { EOFException } to decide if there was a premature EOF. currentPos and filelength are not relative to startPos , thus it's not valid to compare the current pos to the stream length. Ex. I have 128 bytes. I seek 100 bytes into it. The remaining content-length is 28. My file length is not 28 bytes! I read more 10 bytes and the connection unexpectedly closes. The broken premature EOF condition fails to detect the fault because (110 < 28) is false. The correct check is (110 < 100+28). filelength ------------------------ ^---------------- startPos content-length I can file a separate jira for this 1-line fix if you'd like.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Daryn!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Daryn!
          Hide
          Aaron T. Myers added a comment -

          Works for me. Just wanted to make sure it wasn't inadvertent.

          Show
          Aaron T. Myers added a comment - Works for me. Just wanted to make sure it wasn't inadvertent.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2131 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2131/)
          HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2131 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2131/ ) HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2147/)
          HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2147/ ) HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2205/)
          HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2205/ ) HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #239 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/)
          svn merge -c 1330500 from trunk for HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. (Revision 1330504)

          Result = UNSTABLE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330504
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #239 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/ ) svn merge -c 1330500 from trunk for HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. (Revision 1330504) Result = UNSTABLE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330504 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/)
          HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1026 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/ ) HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/)
          HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1061 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/ ) HDFS-3318 . Use BoundedInputStream in ByteRangeInputStream, otherwise, it hangs on transfers >2 GB. Contributed by Daryn Sharp (Revision 1330500) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330500 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ByteRangeInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteRangeInputStream.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Is this also a problem in branch-1?

          Show
          Tsz Wo Nicholas Sze added a comment - Is this also a problem in branch-1?
          Hide
          Robert Joseph Evans added a comment -

          Yes this probably also impacts branch-1.

          Show
          Robert Joseph Evans added a comment - Yes this probably also impacts branch-1.
          Hide
          Daryn Sharp added a comment -

          Yes, on branch-1, >2GB transfers incur a 200s penalty after transfer is complete.

          Show
          Daryn Sharp added a comment - Yes, on branch-1, >2GB transfers incur a 200s penalty after transfer is complete.
          Hide
          Eli Collins added a comment -

          Marking this as an incompatible change as the following code broke distcp from 0.20.2 and 0.21 releases. The code below requires the content length header be present however that field wasn't introduced in trunk until 22 (by HDFS-1085), it came into branch-1 via the YDH merge. Filed HDFS-3671 to fix this.

          -      filelength = (cl == null) ? -1 : Long.parseLong(cl);
          -      in = connection.getInputStream();
          +      if (cl == null) {
          +        throw new IOException(StreamFile.CONTENT_LENGTH+" header is missing");
          +      }
          
          Show
          Eli Collins added a comment - Marking this as an incompatible change as the following code broke distcp from 0.20.2 and 0.21 releases. The code below requires the content length header be present however that field wasn't introduced in trunk until 22 (by HDFS-1085 ), it came into branch-1 via the YDH merge. Filed HDFS-3671 to fix this. - filelength = (cl == null ) ? -1 : Long .parseLong(cl); - in = connection.getInputStream(); + if (cl == null ) { + throw new IOException(StreamFile.CONTENT_LENGTH+ " header is missing" ); + }
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The Content-Length check will be removed by HDFS-3577.

          Show
          Tsz Wo Nicholas Sze added a comment - The Content-Length check will be removed by HDFS-3577 .

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development