[HDFS-770] SocketTimeoutException: timeout while waiting for channel to be ready for read

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.1
• Component/s: datanode, hdfs-client, libhdfs, namenode

Environment

Ubuntu Linux 8.04

• Assignee: Unassigned
• Reporter: Leon Mergen
• Created: 13/Nov/09 08:17
• Updated: 21/Feb/14 16:53

Description

We're having issues with timeouts occurring in our client: for some reason, a timeout of 63000 milliseconds is triggered while writing HDFS data. Since we currently have a single-server setup, this results in our client terminating with a "All datanodes are bad" IOException.

We're running all services, including the client, on our single server, so it cannot be a network error. The load on the client is extremely low during this period: only a few kilobytes a minute were being written around the time the error occured.

After browsing a bit online, a lot of people talk about setting "dfs.datanode.socket.write.timeout" to 0 as a solution for this problem. Due to the low load of our system during this period, however, I do feel this is a real error and a timeout that should not be occurring. I have attached 3 logs of the namenode, datanode and client.

It could be that this is related to http://issues.apache.org/jira/browse/HDFS-693

Any pointers on how I can assist to resolve this issue will be greatly appreciated.

Issue Links

• is related to: HDFS-561 Fix write pipeline READ_TIMEOUT
• is related to: HDFS-693 java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write exceptions were cast when trying to read file via StreamFile.
• relates to: HADOOP-4584 Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode
• relates to: HDFS-1075 Separately configure connect timeouts from read timeouts in data path