Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9393

Hbase does not closing a closed socket resulting in many CLOSE_WAIT

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
    • Fix Version/s: 1.4.0, 1.3.2, 1.2.7, 1.1.12, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.
      Show
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.

      Description

      HBase dose not close a dead connection with the datanode.
      This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port.

      The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT

      [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
      13156
      [root@hd2-region3 ~]# ps -ef |grep 21592
      root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
      hbase 21592 1 17 Aug29 ? 03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...

        Attachments

        1. HBASE-9393-branch-1.patch
          14 kB
          Ashish Singhi
        2. HBASE-9393.v18.patch
          12 kB
          Ashish Singhi
        3. HBASE-9393.v17.patch
          11 kB
          Ashish Singhi
        4. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        5. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        6. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        7. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        8. HBASE-9393.v14.patch
          10 kB
          Ashish Singhi
        9. HBASE-9393.v13.patch
          10 kB
          Ashish Singhi
        10. HBASE-9393.v12.patch
          9 kB
          Ashish Singhi
        11. HBASE-9393.v11.patch
          10 kB
          Ashish Singhi
        12. HBASE-9393.v10.patch
          10 kB
          Ashish Singhi
        13. HBASE-9393.v9.patch
          9 kB
          Ashish Singhi
        14. HBASE-9393.v8.patch
          9 kB
          Ashish Singhi
        15. HBASE-9393.v7.patch
          8 kB
          Ashish Singhi
        16. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        17. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        18. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        19. HBASE-9393.v5.patch
          6 kB
          stack
        20. HBASE-9393.v5.patch
          6 kB
          stack
        21. HBASE-9393.v5.patch
          6 kB
          Ashish Singhi
        22. HBASE-9393.v4.patch
          6 kB
          Ashish Singhi
        23. HBASE-9393.v3.patch
          6 kB
          Ashish Singhi
        24. HBASE-9393.v2.patch
          5 kB
          Ashish Singhi
        25. HBASE-9393.v1.patch
          4 kB
          Ashish Singhi
        26. HBASE-9393.patch
          8 kB
          Ashish Singhi

          Issue Links

            Activity

              People

              • Assignee:
                ashish singhi Ashish Singhi
                Reporter:
                aviz Avi Zrachya
              • Votes:
                1 Vote for this issue
                Watchers:
                49 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: