Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6109

let sync_file_range() system call run in background

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0-alpha1
    • Fix Version/s: 2.5.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Through we passed SYNC_FILE_RANGE_WRITE to sync_file_range, to make it as asynchronous as possible, it still could be blocked, e.g. the os io request queue is full.
      Since we use sync_file_range just as a page cache advisor role it doesn't decide or guarantee the real durability, it would be nice if we could run it in backgroud. At least my test log showed, a few sync_file_range calls still cost tens of ms or more, due to the happened location is in the critical write path(BlockReceiver class), from a upper view, like HBase application, will "hung" tens of ms as well during Hlog syncing.
      Generally speaking, the patch could not improve too much, but, better than before, right ?

        Attachments

        1. HDFS-6109.txt
          9 kB
          Liang Xie
        2. HDFS-6109-v2.txt
          9 kB
          Liang Xie
        3. HDFS-6109-v3.txt
          12 kB
          Liang Xie
        4. HDFS-6109-v4.txt
          9 kB
          Liang Xie

          Activity

            People

            • Assignee:
              xieliang007 Liang Xie
              Reporter:
              xieliang007 Liang Xie
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: