[HDFS-5776] Support 'hedged' reads in DFSClient

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.4.0
• Component/s: hdfs-client
• Hadoop Flags: Reviewed

Release Note

If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc.

This feature is off by default. To enable this feature, set <code>dfs.client.hedged.read.threadpool.size</code> to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client.

Then set <code>dfs.client.hedged.read.threshold.millis</code> to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica.

This feature emits new metrics:

+ hedgedReadOps
+ hedgeReadOpsWin -- how many times the hedged read 'beat' the original read
+ hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum.

• Assignee: Liang Xie
• Reporter: Liang Xie
• Created: 15/Jan/14 03:39
• Updated: 12/May/16 18:18
• Resolved: 24/Feb/14 22:59

Description

This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509

The quorum read ability should be helpful especially to optimize read outliers

we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric).

The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items.

Issue Links

• is depended upon by: HBASE-7509 Enable RS to query a secondary datanode in parallel, if the primary takes too long
• is related to: HDFS-8955 Support 'hedged' write in DFSClient
• relates to: HDFS-6231 DFSClient hangs infinitely if using hedged reads and all eligible datanodes die.
• relates to: HDFS-6450 Support non-positional hedged reads in HDFS