Hadoop Common
  1. Hadoop Common
  2. HADOOP-8135

Add ByteBufferReadable interface to FSDataInputStream

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To prepare for HDFS-2834, it's useful to add an interface to FSDataInputStream (and others inside hdfs) that adds a read(ByteBuffer...) method as follows:

        /**
         * Reads up to buf.remaining() bytes into buf. Callers should use
         * buf.limit(..) to control the size of the desired read.
         * 
         * After the call, buf.position() should be unchanged, and therefore any data
         * can be immediately read from buf.
         * 
         * @param buf
         * @return - the number of bytes available to read from buf
         * @throws IOException
         */
        public int read(ByteBuffer buf) throws IOException;
      
      1. HADOOP-8135.2.patch
        4 kB
        Henry Robinson
      2. HADOOP-8135.patch
        5 kB
        Henry Robinson

        Issue Links

          Activity

          Hide
          Henry Robinson added a comment -

          Here's a patch which adds the interface to FSDataInputStream, and implements it by trying to call on the wrapped stream, and throwing UnsupportedOperationException if nto found.

          Show
          Henry Robinson added a comment - Here's a patch which adds the interface to FSDataInputStream, and implements it by trying to call on the wrapped stream, and throwing UnsupportedOperationException if nto found.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516892/HADOOP-8135.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516892/HADOOP-8135.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Patch looks pretty good to me, Henry. Just a few little nits. +1 once these are addressed:

          1. Please add a javadoc explaining the "buf" param and the IOException for ByteBufferReadable#read.
          2. Please remove the unrelated changes adding @Override annotations to the other methods of FSDataInputStream.

          Even though this patch introduces an interface which as yet has no actual implementation, I'd like to commit it to trunk anyway so that test-patch can be used during reviews on HDFS-2834. If it turns out that HDFS-2834 doesn't get committed in a timely fashion, then I'll revert this patch.

          Any objections to this plan? If not, I'll commit this in a few hours.

          Show
          Aaron T. Myers added a comment - Patch looks pretty good to me, Henry. Just a few little nits. +1 once these are addressed: Please add a javadoc explaining the "buf" param and the IOException for ByteBufferReadable#read. Please remove the unrelated changes adding @Override annotations to the other methods of FSDataInputStream. Even though this patch introduces an interface which as yet has no actual implementation, I'd like to commit it to trunk anyway so that test-patch can be used during reviews on HDFS-2834 . If it turns out that HDFS-2834 doesn't get committed in a timely fashion, then I'll revert this patch. Any objections to this plan? If not, I'll commit this in a few hours.
          Hide
          Todd Lipcon added a comment -
          +   * @return - the number of bytes available to read from buf
          

          style nit: no '-' here

          • maybe worth noting in the javadoc that many FS implementations may throw UnsupportedOperationException, and add that to the javadoc as well
          Show
          Todd Lipcon added a comment - + * @ return - the number of bytes available to read from buf style nit: no '-' here maybe worth noting in the javadoc that many FS implementations may throw UnsupportedOperationException, and add that to the javadoc as well
          Hide
          Henry Robinson added a comment -

          Thanks for the review! Patch updated per comments.

          Show
          Henry Robinson added a comment - Thanks for the review! Patch updated per comments.
          Hide
          Aaron T. Myers added a comment -

          +1

          Show
          Aaron T. Myers added a comment - +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516902/HADOOP-8135.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.ipc.TestRPCCallBenchmark

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516902/HADOOP-8135.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.ipc.TestRPCCallBenchmark +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this. Thanks a lot for the contribution, Hank.

          Show
          Aaron T. Myers added a comment - I've just committed this. Thanks a lot for the contribution, Hank.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1832 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1832/)
          HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556)

          Result = ABORTED
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1832 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1832/ ) HADOOP-8135 . Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1825 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1825/)
          HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1825 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1825/ ) HADOOP-8135 . Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1899 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1899/)
          HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1899 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1899/ ) HADOOP-8135 . Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #973 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/973/)
          HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #973 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/973/ ) HADOOP-8135 . Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1008 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1008/)
          HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1008 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1008/ ) HADOOP-8135 . Add ByteBufferReadable interface to FSDataInputStream. Contributed by Henry Robinson. (Revision 1296556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1296556 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 as well.

          Show
          Todd Lipcon added a comment - Committed to branch-2 as well.

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development