Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2465

Add HDFS support for fadvise readahead and drop-behind

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.23.0, 1.1.0
    • datanode, performance
    • None
    • Reviewed
    • Hide
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
      dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
      dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
      dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
      dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
      Show
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads

    Description

      This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.

      Attachments

        1. hdfs-2465.txt
          16 kB
          Todd Lipcon
        2. hdfs-2465.txt
          16 kB
          Todd Lipcon
        3. hdfs-2465.txt
          16 kB
          Todd Lipcon
        4. hdfs-2465.txt
          16 kB
          Todd Lipcon
        5. HDFS-2465.branch-1.patch
          16 kB
          Brandon Li

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment