Hadoop Common
  1. Hadoop Common
  2. HADOOP-6226

Create a LimitedByteArrayOutputStream that does not expand its buffer on write

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      New LimitedByteArrayOutputStream does not expand buffer on writes.

      Description

      For MAPREDUCE-318, we would benefit from a ByteArrayOutputStream that would support a fixed size buffer that will not expand dynamically on writes.

      1. hadoop-6226-v3.patch
        11 kB
        Jothi Padmanabhan
      2. hadoop-6226-v2.patch
        10 kB
        Jothi Padmanabhan
      3. hadoop-6226-v1.patch
        7 kB
        Jothi Padmanabhan
      4. hadoop-6226.patch
        7 kB
        Jothi Padmanabhan

        Activity

        Hide
        Jothi Padmanabhan added a comment -

        Actually, we will just take care of this inside the Shuffle package itself since the use case seems to be pretty specific

        Show
        Jothi Padmanabhan added a comment - Actually, we will just take care of this inside the Shuffle package itself since the use case seems to be pretty specific
        Hide
        Hong Tang added a comment -

        Actually, the same usage exists in TFile too. I wrote a class called BoundedByteArrayOutputStream which is package private. Might it be useful to promote to a public class in io package?

        Show
        Hong Tang added a comment - Actually, the same usage exists in TFile too. I wrote a class called BoundedByteArrayOutputStream which is package private. Might it be useful to promote to a public class in io package?
        Hide
        Jothi Padmanabhan added a comment -

        Yes, the BoundedByteArrayOutputStream is exactly what we were planning to write too. I am +1 to making it as a public class in io.

        Show
        Jothi Padmanabhan added a comment - Yes, the BoundedByteArrayOutputStream is exactly what we were planning to write too. I am +1 to making it as a public class in io.
        Hide
        Jothi Padmanabhan added a comment -

        Reopening this jira to move BoundedByteArrayOutputStream from tfile Package to io package

        Show
        Jothi Padmanabhan added a comment - Reopening this jira to move BoundedByteArrayOutputStream from tfile Package to io package
        Hide
        Jothi Padmanabhan added a comment -

        Straight forward patch.
        I also added java doc for the public methods

        Show
        Jothi Padmanabhan added a comment - Straight forward patch. I also added java doc for the public methods
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418335/hadoop-6226.patch
        against trunk revision 810329.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418335/hadoop-6226.patch against trunk revision 810329. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/6/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        Fixed a typo in the java doc

        Show
        Jothi Padmanabhan added a comment - Fixed a typo in the java doc
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418338/hadoop-6226-v1.patch
        against trunk revision 810329.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418338/hadoop-6226-v1.patch against trunk revision 810329. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/7/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        Patch that adds a test case as well

        Show
        Jothi Padmanabhan added a comment - Patch that adds a test case as well
        Hide
        Hong Tang added a comment -

        @Jothi, thanks for taking time porting the class to io package and writing unit tests for it. Some comments for consideration:

        • The comment of the test case refers to the class LimitedByteArrayOutputStream.
          +/** Unit tests for LimitedByteArrayOutputStream */
          
        • Should put some random contents in the input byte array. Something like the following (capitalize variable INPUT because it is now static final):
          private static final byte[] INPUT = new byte[SIZE];
          static {
              new Random().nextBytes(INPUT);
          }
          
        • Suggest to write only one byte to test whether an exception is thrown due to limit exceeded.
        • Suggest to test that after reset() and writing INPUT, writing one more byte would cause an exception.
        • Suggest to do reset(SIZE-1) instead of reset(SIZE/2).
        • Suggest to add an assertion (getLimit() == SIZE) after reset() and (getLimit() == SIZE-1) after reset(SIZE-1).
        Show
        Hong Tang added a comment - @Jothi, thanks for taking time porting the class to io package and writing unit tests for it. Some comments for consideration: The comment of the test case refers to the class LimitedByteArrayOutputStream. +/** Unit tests for LimitedByteArrayOutputStream */ Should put some random contents in the input byte array. Something like the following (capitalize variable INPUT because it is now static final): private static final byte [] INPUT = new byte [SIZE]; static { new Random().nextBytes(INPUT); } Suggest to write only one byte to test whether an exception is thrown due to limit exceeded. Suggest to test that after reset() and writing INPUT, writing one more byte would cause an exception. Suggest to do reset(SIZE-1) instead of reset(SIZE/2). Suggest to add an assertion (getLimit() == SIZE) after reset() and (getLimit() == SIZE-1) after reset(SIZE-1).
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418355/hadoop-6226-v2.patch
        against trunk revision 810384.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418355/hadoop-6226-v2.patch against trunk revision 810384. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/9/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        @Hong, thanks for the review.

        Attached a patch that incorporates the review comments

        Show
        Jothi Padmanabhan added a comment - @Hong, thanks for the review. Attached a patch that incorporates the review comments
        Hide
        Hong Tang added a comment -

        The patch looks good. +1.

        Show
        Hong Tang added a comment - The patch looks good. +1.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418358/hadoop-6226-v3.patch
        against trunk revision 810384.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418358/hadoop-6226-v3.patch against trunk revision 810384. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/10/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418358/hadoop-6226-v3.patch
        against trunk revision 810384.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418358/hadoop-6226-v3.patch against trunk revision 810384. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/2/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Jothi!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Jothi!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #8 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/8/)
        . Moves BoundedByteArrayOutputStream from the tfile package to the io package and makes it available to other users (MAPREDUCE-318). Contributed by Jothi Padmanabhan.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #8 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/8/ ) . Moves BoundedByteArrayOutputStream from the tfile package to the io package and makes it available to other users ( MAPREDUCE-318 ). Contributed by Jothi Padmanabhan.
        Hide
        Jothi Padmanabhan added a comment -

        I ran all the Mapreduce and HDFS tests with this jar and they all passed

        Show
        Jothi Padmanabhan added a comment - I ran all the Mapreduce and HDFS tests with this jar and they all passed
        Hide
        Devaraj Das added a comment -

        I committed the hadoop-core*.jar in both the mapreduce and the hdfs projects. The jars has the fixes for HADOOP-6224 and HADOOP-6226.

        Show
        Devaraj Das added a comment - I committed the hadoop-core*.jar in both the mapreduce and the hdfs projects. The jars has the fixes for HADOOP-6224 and HADOOP-6226 .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/11/)
        Checking in the updated common jar that has the updates for and HADOOP-6224.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/11/ ) Checking in the updated common jar that has the updates for and HADOOP-6224 .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #70 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/70/)
        Checking in the updated common jar that has the updates for and HADOOP-6224.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #70 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/70/ ) Checking in the updated common jar that has the updates for and HADOOP-6224 .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #12 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/12/)
        Checking in the updated common jar that has the updates for and HADOOP-6224.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #12 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/12/ ) Checking in the updated common jar that has the updates for and HADOOP-6224 .
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #8 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/8/)

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #8 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/8/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #71 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/71/)
        Checking in the updated common jar that has the updates for and HADOOP-6224.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #71 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/71/ ) Checking in the updated common jar that has the updates for and HADOOP-6224 .
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

          People

          • Assignee:
            Jothi Padmanabhan
            Reporter:
            Jothi Padmanabhan
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development