Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7714

Umbrella for usage of native calls to manage OS cache and readahead

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.24.0
    • Fix Version/s: None
    • Component/s: io, native, performance
    • Labels:
      None

      Description

      Especially in shared HBase/MR situations, management of the OS buffer cache is important. Currently, running a big MR job will evict all of HBase's hot data from cache, causing HBase performance to really suffer. However, caching of the MR input/output is rarely useful, since the datasets tend to be larger than cache and not re-read often enough that the cache is used. Having access to the native calls posix_fadvise and sync_data_range on platforms where they are supported would allow us to do a better job of managing this cache.

        Attachments

        1. hadoop-7714-20s-prelim.txt
          27 kB
          Todd Lipcon
        2. graphs.pdf
          261 kB
          Todd Lipcon
        3. hadoop-7714-2.txt
          39 kB
          Todd Lipcon
        4. 7714-fallocate-20s.patch
          43 kB
          Sriram Rao

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                2 Vote for this issue
                Watchers:
                54 Start watching this issue

                Dates

                • Created:
                  Updated: