[HADOOP-7714] Umbrella for usage of native calls to manage OS cache and readahead

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.24.0
• Component/s: io, native, performance
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 03/Oct/11 17:17
• Updated: 24/Apr/18 01:35

Description

Especially in shared HBase/MR situations, management of the OS buffer cache is important. Currently, running a big MR job will evict all of HBase's hot data from cache, causing HBase performance to really suffer. However, caching of the MR input/output is rarely useful, since the datasets tend to be larger than cache and not re-read often enough that the cache is used. Having access to the native calls posix_fadvise and sync_data_range on platforms where they are supported would allow us to do a better job of managing this cache.

Issue Links

• incorporates: HDFS-2465 Add HDFS support for fadvise readahead and drop-behind
• incorporates: MAPREDUCE-3289 Make use of fadvise in the NM's shuffle handler
• incorporates: MAPREDUCE-4511 Add IFile readahead

Sub-Tasks

• Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class: Closed
• Expose file descriptors from Hadoop-wrapped local FileSystems: Closed
• Using mincore to understand the effect of fadvise in the Linux page cache: Resolved
• Support posix_fallocate in NativeIO and add call from DataNode code: Open