Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-691

Limitation on java.io.InputStream.available()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      java.io.InputStream.available() returns an int which has the max value 2^31-1 = 2GB - 1B. It won't work if the number of available bytes is >= 2GB.

      1. h691_20091028.patch
        0.8 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -
          //DFSClient.DFSInputStream
              public synchronized int available() throws IOException {
                if (closed) {
                  throw new IOException("Stream closed");
                }
                return (int) (getFileLength() - pos);
              }
          

          There are two problems associated with this issue:

          1. If the available bytes is >= 2GB, we will lose the higher bits of the value since the value is casted from long to int.
          2. We may need a new API to get the available bytes as a long. This API is useful with hflush().

          I will fix the first problem in this issue.

          Show
          Tsz Wo Nicholas Sze added a comment - //DFSClient.DFSInputStream public synchronized int available() throws IOException { if (closed) { throw new IOException( "Stream closed" ); } return ( int ) (getFileLength() - pos); } There are two problems associated with this issue: If the available bytes is >= 2GB, we will lose the higher bits of the value since the value is casted from long to int. We may need a new API to get the available bytes as a long. This API is useful with hflush(). I will fix the first problem in this issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h691_20091028.patch: check limit before return.

          No new test is added since the change is simple and there is no easy way to implement a new test.

          Show
          Tsz Wo Nicholas Sze added a comment - h691_20091028.patch: check limit before return. No new test is added since the change is simple and there is no easy way to implement a new test.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423492/h691_20091028.patch
          against trunk revision 830003.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423492/h691_20091028.patch against trunk revision 830003. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/85/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423492/h691_20091028.patch
          against trunk revision 830003.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423492/h691_20091028.patch against trunk revision 830003. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/64/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The failure of TestBlockReport was not related to this. See HDFS-733.

          Show
          Tsz Wo Nicholas Sze added a comment - The failure of TestBlockReport was not related to this. See HDFS-733 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this to 0.21 and above .

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this to 0.21 and above .
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/86/)
          . Fix an overflow error in DFSClient.DFSInputStream.available().

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/86/ ) . Fix an overflow error in DFSClient.DFSInputStream.available().
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #124 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/124/)
          . Fix an overflow error in DFSClient.DFSInputStream.available().

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #124 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/124/ ) . Fix an overflow error in DFSClient.DFSInputStream.available().
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #100 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/100/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #72 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/72/ )

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development