[HBASE-11833] Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.98.0
• Component/s: regionserver

Environment

RHEL 6.3 -HDP 2.1 -6 RegionServers/Datanode -18T per node -3108Regions

• Assignee: Unassigned
• Reporter: steven xu
• Created: 27/Aug/14 03:02
• Updated: 17/Jun/22 17:20
• Resolved: 27/Aug/14 20:06

Description

HBase dose not close a dead connection with the datanode.
This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port:50010.
After I restart all RSs, the count of CLOSE_WAIT will increase always.
$ netstat -an|grep CLOSE_WAIT|wc -l
2545

1. netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
   2545
2. ps -ef|grep 6569
   hbase 6569 6556 21 Aug25 ? 09:52:33 /opt/jdk1.6.0_25/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC

I aslo have reviewed these issues:
HBASE-9393
HDFS-5671
HDFS-1836
I found HBase 0.98/Hadoop 2.4.0 I uesed which source codes are not different from these patches.
But I donot understand why HBase 0.98/Hadoop 2.4.0 also have this isssue. Please check. Thanks a lot.