Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4184

Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.

      When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
      When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
      When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads, and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
      and so on... 
      

      Current we can only set these feature global in datanode,we should set these feature per session.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                binlijin Lijin Bin
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: