[HDFS-7005] DFS input streams do not timeout

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.0, 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: hdfs-client
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 05/Sep/14 20:18
• Updated: 08/Jun/16 02:50
• Resolved: 08/Sep/14 19:51

Description

Input streams lost their timeout. The problem appears to be DFSClient#newConnectedPeer does not set the read timeout. During a temporary network interruption the server will close the socket, unbeknownst to the client host, which blocks on a read forever.

The results are dire. Services such as the RM, JHS, NMs, oozie servers, etc all need to be restarted to recover - unless you want to wait many hours for the tcp stack keepalive to detect the broken socket.

Issue Links

• is broken by: HDFS-5810 Unify mmap cache and short-circuit file descriptor cache
• is duplicated by: ACCUMULO-3396 HDFS reads are hanging
• is related to: HDFS-7608 hdfs dfsclient newConnectedPeer has no write timeout
• relates to: HBASE-15900 RS stuck in get lock of HStore