[HDFS-1836] Thousand of CLOSE_WAIT socket

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.2
• Fix Version/s: 0.20.3, 0.20.205.0
• Component/s: hdfs-client

Environment

Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)

• Hadoop Flags: Reviewed
• Assignee: Bharath Mundlapudi
• Reporter: Dennis Cheung
• Created: 14/Apr/11 07:40
• Updated: 19/Oct/11 00:25
• Resolved: 15/Jun/11 00:04

Description

$ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT
4471

It is better if everything runs normal.
However, from time to time there are some "DataStreamer Exception: java.net.SocketTimeoutException" and "DFSClient.processDatanodeError(2507) | Error Recovery for" can be found from log file and the number of CLOSE_WAIT socket just keep increasing

The CLOSE_WAIT handles may remain for hours and days; then "Too many open file" some day.