Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2465

Add HDFS support for fadvise readahead and drop-behind

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0, 1.1.0
    • Component/s: datanode, performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
      dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
      dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
      dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
      dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
      Show
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads

      Description

      This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.

      1. hdfs-2465.txt
        16 kB
        Todd Lipcon
      2. hdfs-2465.txt
        16 kB
        Todd Lipcon
      3. hdfs-2465.txt
        16 kB
        Todd Lipcon
      4. hdfs-2465.txt
        16 kB
        Todd Lipcon
      5. HDFS-2465.branch-1.patch
        16 kB
        Brandon Li

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is part of HADOOP-7714 [ HADOOP-7714 ]
          Todd Lipcon made changes -
          Link This issue is blocked by HADOOP-7753 [ HADOOP-7753 ]
          Todd Lipcon made changes -
          Attachment hdfs-2465.txt [ 12499490 ]
          Todd Lipcon made changes -
          Attachment hdfs-2465.txt [ 12501177 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Component/s performance [ 12316501 ]
          Todd Lipcon made changes -
          Attachment hdfs-2465.txt [ 12501194 ]
          Todd Lipcon made changes -
          Attachment hdfs-2465.txt [ 12501334 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
          dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
          dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
          dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
          dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
          Fix Version/s 0.23.0 [ 12315571 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-3228 [ HDFS-3228 ]
          Brandon Li made changes -
          Attachment HDFS-2465.branch-1.patch [ 12535917 ]
          Suresh Srinivas made changes -
          Fix Version/s 1.2.0 [ 12321657 ]
          Brandon Li made changes -
          Link This issue is related to HDFS-2751 [ HDFS-2751 ]
          Suresh Srinivas made changes -
          Fix Version/s 1.1.0 [ 12317959 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          binlijin made changes -
          Link This issue is related too HDFS-4184 [ HDFS-4184 ]
          Gavin made changes -
          Link This issue is related to HDFS-4184 [ HDFS-4184 ]
          Gavin made changes -
          Link This issue is related to HDFS-4184 [ HDFS-4184 ]

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development