HBase
  1. HBase
  2. HBASE-9393

Hbase does not closing a closed socket resulting in many CLOSE_WAIT

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.94.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions

      Description

      HBase dose not close a dead connection with the datanode.
      This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port.

      The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT

      [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
      13156
      [root@hd2-region3 ~]# ps -ef |grep 21592
      root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
      hbase 21592 1 17 Aug29 ? 03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Avi Zrachya
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:

              Development