Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7714

Umbrella for usage of native calls to manage OS cache and readahead

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.24.0
    • None
    • io, native, performance
    • None

    Description

      Especially in shared HBase/MR situations, management of the OS buffer cache is important. Currently, running a big MR job will evict all of HBase's hot data from cache, causing HBase performance to really suffer. However, caching of the MR input/output is rarely useful, since the datasets tend to be larger than cache and not re-read often enough that the cache is used. Having access to the native calls posix_fadvise and sync_data_range on platforms where they are supported would allow us to do a better job of managing this cache.

      Attachments

        1. 7714-fallocate-20s.patch
          43 kB
          Sriram Rao
        2. hadoop-7714-2.txt
          39 kB
          Todd Lipcon
        3. graphs.pdf
          261 kB
          Todd Lipcon
        4. hadoop-7714-20s-prelim.txt
          27 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tlipcon Todd Lipcon

            Dates

              Created:
              Updated:

              Slack

                Issue deployment