Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7694

FSDataInputStream should support "unbuffer"

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.7.0, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      For applications that have many open HDFS (or other Hadoop filesystem) files, it would be useful to have an API to clear readahead buffers and sockets. This could be added to the existing APIs as an optional interface, in much the same way as we added setReadahead / setDropBehind / etc.

      1. HDFS-7694.001.patch
        13 kB
        Colin P. McCabe
      2. HDFS-7694.002.patch
        12 kB
        Colin P. McCabe
      3. HDFS-7694.003.patch
        13 kB
        Colin P. McCabe
      4. HDFS-7694.004.patch
        12 kB
        Colin P. McCabe
      5. HDFS-7694.005.patch
        12 kB
        Colin P. McCabe

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695104/HDFS-7694.001.patch
          against trunk revision caf7298.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9362//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9362//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695104/HDFS-7694.001.patch against trunk revision caf7298. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9362//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9362//console This message is automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Hi Colin, could you please re-base the patch?
          One question, in what cases, user needs to unbuffer instead of closing the stream?

          Show
          hitliuyi Yi Liu added a comment - Hi Colin, could you please re-base the patch? One question, in what cases, user needs to unbuffer instead of closing the stream?
          Hide
          cmccabe Colin P. McCabe added a comment -

          One question, in what cases, user needs to unbuffer instead of closing the stream?

          Good question. The main answer is that re-opening a stream will cause a getBlockLocations RPC to the NameNode. Some applications cache a lot of open streams in order to avoid generating a lot of NameNode traffic. HBase is one, Impala is another. This change is a really easy way to let those applications save memory without generating a lot of RPC load on the NN.

          Show
          cmccabe Colin P. McCabe added a comment - One question, in what cases, user needs to unbuffer instead of closing the stream? Good question. The main answer is that re-opening a stream will cause a getBlockLocations RPC to the NameNode. Some applications cache a lot of open streams in order to avoid generating a lot of NameNode traffic. HBase is one, Impala is another. This change is a really easy way to let those applications save memory without generating a lot of RPC load on the NN.
          Hide
          cmccabe Colin P. McCabe added a comment -

          rebased patch

          Show
          cmccabe Colin P. McCabe added a comment - rebased patch
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696959/HDFS-7694.002.patch
          against trunk revision 9d91069.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9452//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9452//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696959/HDFS-7694.002.patch against trunk revision 9d91069. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9452//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9452//console This message is automatically generated.
          Hide
          stack stack added a comment -

          CanUnbuffer ain't too pretty. Unbufferable is about as ugly. Its fine I suppose as is.

          In DFSIS#unbuffer, should we be resetting data members back to zero, etc?

          In testOpenManyFilesViaTcp, we assert we can read but is there a reason why we would not be able to that unbuffer enables? (pardon if dumb question)

          Show
          stack stack added a comment - CanUnbuffer ain't too pretty. Unbufferable is about as ugly. Its fine I suppose as is. In DFSIS#unbuffer, should we be resetting data members back to zero, etc? In testOpenManyFilesViaTcp, we assert we can read but is there a reason why we would not be able to that unbuffer enables? (pardon if dumb question)
          Hide
          cmccabe Colin P. McCabe added a comment -

          CanUnbuffer ain't too pretty. Unbufferable is about as ugly. Its fine I suppose as is.

          It's consistent with our other "input stream extension" interfaces such as Syncable, CanSetReadahead, etc. The problem is that we can't add the new APIs to FSInputStream, or else we'd break a bunch of non-HDFS streams (in and out of the tree) that don't implement the new API. I guess Java is adding default implementations for interface functions in some future version... too bad we're not there yet.

          l In DFSIS#unbuffer, should we be resetting data members back to zero, etc?

          I'm not sure what else we'd reset. This isn't changing the closed state, it's not a seek so the pos is not affected, it's not changing the cachingStrategy or fileEncryptionInfo... we certainly don't want to clear the block location info because then we need to do an RPC to the NN to get it again...

          Actually I do see one thing we should change. We should set blockEnd to -1. Otherwise, seek may attempt to use blockReader even though it's null. It seems like this is also a problem in closeCurrentBlockReader. And let me add a seek after the unbuffer in testUnbufferClosesSockets to make sure that this doesn't regress.

          In testOpenManyFilesViaTcp, we assert we can read but is there a reason why we would not be able to that unbuffer enables? (pardon if dumb question)

          Not a dumb question at all. What I was testing here was that opening a lot of files didn't consume too many resources. In my local test environment, I increased NUM_OPENS to be a really big number... I didn't want to burden Jenkins too much, though. testUnbufferClosesSockets is a more "direct" and straightforward test than testOpenManyFilesViaTcp... the latter is perhaps more of a stress test.

          Show
          cmccabe Colin P. McCabe added a comment - CanUnbuffer ain't too pretty. Unbufferable is about as ugly. Its fine I suppose as is. It's consistent with our other "input stream extension" interfaces such as Syncable , CanSetReadahead , etc. The problem is that we can't add the new APIs to FSInputStream , or else we'd break a bunch of non-HDFS streams (in and out of the tree) that don't implement the new API. I guess Java is adding default implementations for interface functions in some future version... too bad we're not there yet. l In DFSIS#unbuffer, should we be resetting data members back to zero, etc? I'm not sure what else we'd reset. This isn't changing the closed state, it's not a seek so the pos is not affected, it's not changing the cachingStrategy or fileEncryptionInfo ... we certainly don't want to clear the block location info because then we need to do an RPC to the NN to get it again... Actually I do see one thing we should change. We should set blockEnd to -1. Otherwise, seek may attempt to use blockReader even though it's null . It seems like this is also a problem in closeCurrentBlockReader . And let me add a seek after the unbuffer in testUnbufferClosesSockets to make sure that this doesn't regress. In testOpenManyFilesViaTcp, we assert we can read but is there a reason why we would not be able to that unbuffer enables? (pardon if dumb question) Not a dumb question at all. What I was testing here was that opening a lot of files didn't consume too many resources. In my local test environment, I increased NUM_OPENS to be a really big number... I didn't want to burden Jenkins too much, though. testUnbufferClosesSockets is a more "direct" and straightforward test than testOpenManyFilesViaTcp ... the latter is perhaps more of a stress test.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I filed HDFS-7744 for the existing "seek after setReadahead" bug. It's not good form to combine a new feature with a bugfix.

          Show
          cmccabe Colin P. McCabe added a comment - I filed HDFS-7744 for the existing "seek after setReadahead" bug. It's not good form to combine a new feature with a bugfix.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697115/HDFS-7694.003.patch
          against trunk revision 4c48432.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ipc.TestCallQueueManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9466//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697115/HDFS-7694.003.patch against trunk revision 4c48432. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ipc.TestCallQueueManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9466//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9466//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          rebase on HDFS-7744 now that it's in

          Show
          cmccabe Colin P. McCabe added a comment - rebase on HDFS-7744 now that it's in
          Hide
          hitliuyi Yi Liu added a comment -

          The latest patch looks good to me, thanks Colin. One comment:
          could we verify that we still read successfully and verify the read content is correct after we do unbuffer?

          Show
          hitliuyi Yi Liu added a comment - The latest patch looks good to me, thanks Colin. One comment: could we verify that we still read successfully and verify the read content is correct after we do unbuffer ?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697877/HDFS-7694.004.patch
          against trunk revision e9d26fe.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697877/HDFS-7694.004.patch against trunk revision e9d26fe. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9522//console This message is automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          add a test of reading after calling unbuffer()

          Show
          cmccabe Colin P. McCabe added a comment - add a test of reading after calling unbuffer()
          Hide
          hitliuyi Yi Liu added a comment -

          +1 pending Jenkins. Thanks Colin.

          Show
          hitliuyi Yi Liu added a comment - +1 pending Jenkins. Thanks Colin.
          Hide
          umamaheswararao Uma Maheswara Rao G added a comment -

          Yes, this should be useful. +1

          Show
          umamaheswararao Uma Maheswara Rao G added a comment - Yes, this should be useful. +1
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698303/HDFS-7694.005.patch
          against trunk revision 8a54384.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9555//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9555//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698303/HDFS-7694.005.patch against trunk revision 8a54384. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9555//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9555//console This message is automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7091 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7091/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7091 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7091/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/103/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/103/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #837 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/837/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #837 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/837/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2035 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2035/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2035 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2035/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/104/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/104/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2054 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2054/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2054 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2054/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/)
          HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/ ) HDFS-7694 . FSDataInputStream should support "unbuffer" (cmccabe) (cmccabe: rev 6b39ad0865cb2a7960dd59d68178f0bf28865ce2) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CanUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/PeerCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestUnbuffer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Is there compatibility concern for backporting this to 2.6 branch ?

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Is there compatibility concern for backporting this to 2.6 branch ? Thanks
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Colin P. McCabe Junping Du:
          Patch applies on branch-2.6 cleanly.
          Can you commit to branch-2.6 ?

          This would benefit HBASE-9393

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Colin P. McCabe Junping Du : Patch applies on branch-2.6 cleanly. Can you commit to branch-2.6 ? This would benefit HBASE-9393 Thanks
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Colin P. McCabe Junping Du:
          Patch applies on branch-2.6 cleanly.
          Can you commit to branch-2.6 ?

          This would benefit HBASE-9393

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Colin P. McCabe Junping Du : Patch applies on branch-2.6 cleanly. Can you commit to branch-2.6 ? This would benefit HBASE-9393 Thanks
          Hide
          djp Junping Du added a comment -

          Is there compatibility concern for backporting this to 2.6 branch?

          I don't think so, after checking the patch and from description above "This could be added to the existing APIs as an optional interface." Colin P. McCabe, what do you think?

          Show
          djp Junping Du added a comment - Is there compatibility concern for backporting this to 2.6 branch? I don't think so, after checking the patch and from description above "This could be added to the existing APIs as an optional interface." Colin P. McCabe , what do you think?
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hi, Junping Du. This change is compatible, since people are not expected to be subclassing FSDataInputStream. So it seems fine to backport to 2.6, if the maintainers of that branch think it will be useful there.

          Show
          cmccabe Colin P. McCabe added a comment - Hi, Junping Du . This change is compatible, since people are not expected to be subclassing FSDataInputStream . So it seems fine to backport to 2.6, if the maintainers of that branch think it will be useful there.
          Hide
          djp Junping Du added a comment -

          Thanks Colin P. McCabe for confirmation. I will cherry-pick this patch to branch-2.6 later when build failure (caused by HADOOP-12715) is figured out.

          Show
          djp Junping Du added a comment - Thanks Colin P. McCabe for confirmation. I will cherry-pick this patch to branch-2.6 later when build failure (caused by HADOOP-12715 ) is figured out.
          Hide
          djp Junping Du added a comment -

          I have cherry-pick it to branch-2.6.

          Show
          djp Junping Du added a comment - I have cherry-pick it to branch-2.6.
          Hide
          cnauroth Chris Nauroth added a comment -

          As per discussion on HADOOP-12805 with the HBase community, I'd like to make CanUnbuffer a Public interface. I'd appreciate if the original contributors here could help chime in on HADOOP-12805. Thanks!

          Show
          cnauroth Chris Nauroth added a comment - As per discussion on HADOOP-12805 with the HBase community, I'd like to make CanUnbuffer a Public interface. I'd appreciate if the original contributors here could help chime in on HADOOP-12805 . Thanks!

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development