[HDFS-6286] adding a timeout setting for local read io

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.0, 3.0.0-alpha1
• Component/s: hdfs-client
• Assignee: Liang Xie
• Reporter: Liang Xie
• Created: 25/Apr/14 07:10
• Updated: 12/May/16 18:13
• Resolved: 28/May/14 03:34

Description

Currently, if a write or remote read requested into a sick disk, DFSClient.hdfsTimeout could help the caller have a guaranteed time cost to return back. but it doesn't work on local read. Take HBase scan for example,
DFSInputStream.read -> readWithStrategy -> readBuffer -> BlockReaderLocal.read -> dataIn.read -> FileChannelImpl.read
if it hits a bad disk, the low read io probably takes tens of seconds, and what's worse is, the "DFSInputStream.read" hold a lock always.
Per my knowledge, there's no good mechanism to cancel a running read io(Please correct me if it's wrong), so my opinion is adding a future around the read request, and we could set a timeout there, if the threshold reached, we can add the local node into deadnode probably...
Any thought?

Issue Links

• depends upon: HDFS-6450 Support non-positional hedged reads in HDFS
• is superceded by: HDFS-6450 Support non-positional hedged reads in HDFS