[HDFS-2465] Add HDFS support for fadvise readahead and drop-behind

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0, 1.1.0
• Component/s: datanode, performance
• Target Version/s: 0.23.0
• Hadoop Flags: Reviewed

Release Note

HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads

• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 18/Oct/11 02:37
• Updated: 02/May/13 02:30
• Resolved: 28/Oct/11 22:21

Description

This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.

Issue Links

• is blocked by: HADOOP-7753 Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class
• is part of: HADOOP-7714 Umbrella for usage of native calls to manage OS cache and readahead
• is related to: HDFS-2751 Datanode drops OS cache behind reads even for short reads
• is related to: HDFS-4184 Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
• relates to: HDFS-3228 Use fadvise in local read path