Hadoop Common
  1. Hadoop Common
  2. HADOOP-7714

Umbrella for usage of native calls to manage OS cache and readahead

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.24.0
    • Fix Version/s: None
    • Component/s: io, native, performance
    • Labels:
      None

      Description

      Especially in shared HBase/MR situations, management of the OS buffer cache is important. Currently, running a big MR job will evict all of HBase's hot data from cache, causing HBase performance to really suffer. However, caching of the MR input/output is rarely useful, since the datasets tend to be larger than cache and not re-read often enough that the cache is used. Having access to the native calls posix_fadvise and sync_data_range on platforms where they are supported would allow us to do a better job of managing this cache.

      1. 7714-fallocate-20s.patch
        43 kB
        Sriram Rao
      2. graphs.pdf
        261 kB
        Todd Lipcon
      3. hadoop-7714-2.txt
        39 kB
        Todd Lipcon
      4. hadoop-7714-20s-prelim.txt
        27 kB
        Todd Lipcon

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              2 Vote for this issue
              Watchers:
              56 Start watching this issue

              Dates

              • Created:
                Updated:

                Development