Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.4.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc.

      This feature is off by default. To enable this feature, set <code>dfs.client.hedged.read.threadpool.size</code> to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client.

      Then set <code>dfs.client.hedged.read.threshold.millis</code> to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica.

      This feature emits new metrics:

      + hedgedReadOps
      + hedgeReadOpsWin -- how many times the hedged read 'beat' the original read
      + hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum.
      Show
      If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set <code>dfs.client.hedged.read.threadpool.size</code> to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client. Then set <code>dfs.client.hedged.read.threshold.millis</code> to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedReadOps + hedgeReadOpsWin -- how many times the hedged read 'beat' the original read + hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum.

      Description

      This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509

      The quorum read ability should be helpful especially to optimize read outliers

      we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric).

      The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items.

        Attachments

        1. HDFS-5776.txt
          29 kB
          Liang Xie
        2. HDFS-5776-v2.txt
          29 kB
          Liang Xie
        3. HDFS-5776-v3.txt
          30 kB
          Liang Xie
        4. HDFS-5776-v4.txt
          30 kB
          Liang Xie
        5. HDFS-5776-v5.txt
          29 kB
          Liang Xie
        6. HDFS-5776-v6.txt
          30 kB
          Liang Xie
        7. HDFS-5776-v7.txt
          29 kB
          Liang Xie
        8. HDFS-5776-v8.txt
          30 kB
          Liang Xie
        9. HDFS-5776-v9.txt
          31 kB
          Liang Xie
        10. HDFS-5776-v10.txt
          31 kB
          Liang Xie
        11. HDFS-5776-v11.txt
          31 kB
          Liang Xie
        12. HDFS-5776-v12.txt
          31 kB
          stack
        13. HDFS-5776-v12.txt
          31 kB
          stack
        14. HDFS-5776-v13.wip.txt
          31 kB
          stack
        15. HDFS-5776-v14.txt
          30 kB
          stack
        16. HDFS-5776-v15.txt
          32 kB
          stack
        17. HDFS-5776-v17.txt
          33 kB
          stack
        18. HDFS-5776-v17.txt
          33 kB
          stack
        19. HDFS-5776v18.txt
          34 kB
          stack
        20. HDFS-5776v21.txt
          35 kB
          stack
        21. HDFS-5776v21-branch2.txt
          34 kB
          stack

          Issue Links

            Activity

              People

              • Assignee:
                xieliang007 Liang Xie
                Reporter:
                xieliang007 Liang Xie
              • Votes:
                0 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: