Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2465

Add HDFS support for fadvise readahead and drop-behind

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0, 1.1.0
    • Component/s: datanode, performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
      dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
      dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
      dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
      dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
      Show
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads

      Description

      This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.

      1. hdfs-2465.txt
        16 kB
        Todd Lipcon
      2. hdfs-2465.txt
        16 kB
        Todd Lipcon
      3. hdfs-2465.txt
        16 kB
        Todd Lipcon
      4. hdfs-2465.txt
        16 kB
        Todd Lipcon
      5. HDFS-2465.branch-1.patch
        16 kB
        Brandon Li

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Looked at the patch, think this is a reasonable first DN-wide heuristic. Comments follow:

          • There isn't any config that enables syncBehindWrites in BlockerRecevier.
          • Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead.
          • In BlockerSender#sendBlock by the 1st fadvise a comment like "advice applies to the entire file" would help
          • Nit: add _BYTES to LONG_READ_THRESHOLD and CACHE_DROP_INTERVAL.

          Otherwise looks great.

          Show
          Eli Collins added a comment - Looked at the patch, think this is a reasonable first DN-wide heuristic. Comments follow: There isn't any config that enables syncBehindWrites in BlockerRecevier. Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead. In BlockerSender#sendBlock by the 1st fadvise a comment like "advice applies to the entire file" would help Nit: add _BYTES to LONG_READ_THRESHOLD and CACHE_DROP_INTERVAL. Otherwise looks great.
          Hide
          Todd Lipcon added a comment -

          There isn't any config that enables syncBehindWrites in BlockerRecevier

          Woops! good catch. Lost that somewhere along the line.

          Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead.

          I don't think so. HDFS doesn't really do readahead except for filling up the TCP window. But submitting very large (multi-MB)readahead to the OS lets it schedule big sequential reads. So, I don't think there's any real correlation. To set this tunable, I'd probably do some mental math like: "well, I probably have 20 files being read at once, so 8M of readahead on each of those will end up being 160M of RAM used." Having to multiply by other factors is harder to follow.

          Fixed the other nits. New patch momentarily.

          Show
          Todd Lipcon added a comment - There isn't any config that enables syncBehindWrites in BlockerRecevier Woops! good catch. Lost that somewhere along the line. Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead. I don't think so. HDFS doesn't really do readahead except for filling up the TCP window. But submitting very large (multi-MB)readahead to the OS lets it schedule big sequential reads. So, I don't think there's any real correlation. To set this tunable, I'd probably do some mental math like: "well, I probably have 20 files being read at once, so 8M of readahead on each of those will end up being 160M of RAM used." Having to multiply by other factors is harder to follow. Fixed the other nits. New patch momentarily.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501177/hdfs-2465.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed the unit tests build

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501177/hdfs-2465.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          looks like the test-patch build ran before HADOOP-7753 was integrated in the snapshot build. I'll wait half an hour or so and retrigger this test-patch. Meanwhile, should be good to review.

          Show
          Todd Lipcon added a comment - looks like the test-patch build ran before HADOOP-7753 was integrated in the snapshot build. I'll wait half an hour or so and retrigger this test-patch. Meanwhile, should be good to review.
          Hide
          Eli Collins added a comment -

          I don't think so. HDFS doesn't really do readahead except for filling up the TCP window

          I was thinking that filling up the dfs packet is essentially readahead since we'll read the whole 64kb (or whatever is configured) regardless of how much the client asks for but yea in this context that's not relevant (eg HBase won't issue such small reads).

          +1 to your latest patch. (Nit: add spaces in "1024*1024" no need to generate a new patch)

          Show
          Eli Collins added a comment - I don't think so. HDFS doesn't really do readahead except for filling up the TCP window I was thinking that filling up the dfs packet is essentially readahead since we'll read the whole 64kb (or whatever is configured) regardless of how much the client asks for but yea in this context that's not relevant (eg HBase won't issue such small reads). +1 to your latest patch. (Nit: add spaces in "1024*1024" no need to generate a new patch)
          Hide
          Todd Lipcon added a comment -

          reuploading patch to trigger build. Also fixed the style nit Eli mentioned above.

          Show
          Todd Lipcon added a comment - reuploading patch to trigger build. Also fixed the style nit Eli mentioned above.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501194/hdfs-2465.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          org.apache.hadoop.hdfs.TestSmallBlock
          org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.TestFileCreation
          org.apache.hadoop.hdfs.TestFileCreationClient
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
          org.apache.hadoop.hdfs.TestSetrepIncreasing
          org.apache.hadoop.hdfs.TestPread

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501194/hdfs-2465.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestPread +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Had forgotten to deal with the simulated FSDataset in the previous patch. New patch checks whether the block stream is a FileInputStream before attempting to get its file descriptor.

          Show
          Todd Lipcon added a comment - Had forgotten to deal with the simulated FSDataset in the previous patch. New patch checks whether the block stream is a FileInputStream before attempting to get its file descriptor.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501334/hdfs-2465.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501334/hdfs-2465.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1487//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Failing tests seem unrelated. I ran the webhdfs tests locally and they passed. TestHDFSCLI fails on my local box, but does so on trunk as well for me.

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          Unfortunately it's not really possible to unit test this. I ran through several terasorts on a small test cluster with this patch and verified that it helps performance and didn't cause stability issues. I've been testing with very similar patches for a couple weeks and seen no issues. It also is off-by-default.

          Given Eli gave a +1 earlier, I will wait a few hours and commit this if there are no further comments.

          Show
          Todd Lipcon added a comment - Failing tests seem unrelated. I ran the webhdfs tests locally and they passed. TestHDFSCLI fails on my local box, but does so on trunk as well for me. -1 tests included. The patch doesn't appear to include any new or modified tests. Unfortunately it's not really possible to unit test this. I ran through several terasorts on a small test cluster with this patch and verified that it helps performance and didn't cause stability issues. I've been testing with very similar patches for a couple weeks and seen no issues. It also is off-by-default. Given Eli gave a +1 earlier, I will wait a few hours and commit this if there are no further comments.
          Hide
          Eli Collins added a comment -

          +1 latest patch lgtm

          Show
          Eli Collins added a comment - +1 latest patch lgtm
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and 23. Thanks for the review, Eli.

          Show
          Todd Lipcon added a comment - Committed to trunk and 23. Thanks for the review, Eli.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1216 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1216/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1216 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1216/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #105 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/105/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #105 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/105/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1269/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1269/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1193 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1193/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1193 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1193/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190626 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/)
          HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/ ) HDFS-2465 . Add HDFS support for fadvise readahead and drop-behind. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190625 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          Hide
          Brandon Li added a comment -

          Back port it to branch-1.

          Show
          Brandon Li added a comment - Back port it to branch-1.
          Hide
          Suresh Srinivas added a comment -

          +! for the branch-1 patch. Please comment on the tests you have run.

          Show
          Suresh Srinivas added a comment - +! for the branch-1 patch. Please comment on the tests you have run.
          Hide
          Brandon Li added a comment -

          Thanks. I ran teragen/terasort/teravalidate/TestDFSIO tests.

          Show
          Brandon Li added a comment - Thanks. I ran teragen/terasort/teravalidate/TestDFSIO tests.
          Hide
          Suresh Srinivas added a comment -

          This time, going from +! to +1

          Show
          Suresh Srinivas added a comment - This time, going from +! to +1
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to branch-1

          Show
          Suresh Srinivas added a comment - I committed the patch to branch-1

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development