Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7714

Umbrella for usage of native calls to manage OS cache and readahead

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.24.0
    • None
    • io, native, performance
    • None

    Description

      Especially in shared HBase/MR situations, management of the OS buffer cache is important. Currently, running a big MR job will evict all of HBase's hot data from cache, causing HBase performance to really suffer. However, caching of the MR input/output is rarely useful, since the datasets tend to be larger than cache and not re-read often enough that the cache is used. Having access to the native calls posix_fadvise and sync_data_range on platforms where they are supported would allow us to do a better job of managing this cache.

      Attachments

        1. 7714-fallocate-20s.patch
          43 kB
          Sriram Rao
        2. graphs.pdf
          261 kB
          Todd Lipcon
        3. hadoop-7714-2.txt
          39 kB
          Todd Lipcon
        4. hadoop-7714-20s-prelim.txt
          27 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tlipcon Todd Lipcon
              Votes:
              2 Vote for this issue
              Watchers:
              53 Start watching this issue

              Dates

                Created:
                Updated: