Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11833

Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.98.0
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    • Environment:

      RHEL 6.3 -HDP 2.1 -6 RegionServers/Datanode -18T per node -3108Regions

      Description

      HBase dose not close a dead connection with the datanode.
      This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port:50010.
      After I restart all RSs, the count of CLOSE_WAIT will increase always.
      $ netstat -an|grep CLOSE_WAIT|wc -l
      2545

      1. netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
        2545
      2. ps -ef|grep 6569
        hbase 6569 6556 21 Aug25 ? 09:52:33 /opt/jdk1.6.0_25/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC

      I aslo have reviewed these issues:
      HBASE-9393
      HDFS-5671
      HDFS-1836
      I found HBase 0.98/Hadoop 2.4.0 I uesed which source codes are not different from these patches.
      But I donot understand why HBase 0.98/Hadoop 2.4.0 also have this isssue. Please check. Thanks a lot.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevenxu steven xu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: