Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9393

Region Server fails to properly close socket resulting in many CLOSE_WAIT to Data Nodes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
    • 1.4.0, 1.3.2, 1.1.12, 2.0.0, 1.2.7
    • None
    • None
    • Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions

    • Reviewed
    • Hide
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.
      Show
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.

    Description

      HBase dose not close a dead connection with the datanode.
      This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port.

      The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT

      [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
      13156
      [root@hd2-region3 ~]# ps -ef |grep 21592
      root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
      hbase 21592 1 17 Aug29 ? 03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...

      Attachments

        1. HBASE-9393.patch
          8 kB
          Ashish Singhi
        2. HBASE-9393.v1.patch
          4 kB
          Ashish Singhi
        3. HBASE-9393.v10.patch
          10 kB
          Ashish Singhi
        4. HBASE-9393.v11.patch
          10 kB
          Ashish Singhi
        5. HBASE-9393.v12.patch
          9 kB
          Ashish Singhi
        6. HBASE-9393.v13.patch
          10 kB
          Ashish Singhi
        7. HBASE-9393.v14.patch
          10 kB
          Ashish Singhi
        8. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        9. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        10. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        11. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        12. HBASE-9393.v17.patch
          11 kB
          Ashish Singhi
        13. HBASE-9393.v18.patch
          12 kB
          Ashish Singhi
        14. HBASE-9393.v2.patch
          5 kB
          Ashish Singhi
        15. HBASE-9393.v3.patch
          6 kB
          Ashish Singhi
        16. HBASE-9393.v4.patch
          6 kB
          Ashish Singhi
        17. HBASE-9393.v5.patch
          6 kB
          Michael Stack
        18. HBASE-9393.v5.patch
          6 kB
          Michael Stack
        19. HBASE-9393.v5.patch
          6 kB
          Ashish Singhi
        20. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        21. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        22. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        23. HBASE-9393.v7.patch
          8 kB
          Ashish Singhi
        24. HBASE-9393.v8.patch
          9 kB
          Ashish Singhi
        25. HBASE-9393.v9.patch
          9 kB
          Ashish Singhi
        26. HBASE-9393-branch-1.patch
          14 kB
          Ashish Singhi

        Issue Links

          Activity

            People

              ashish singhi Ashish Singhi
              aviz Avi Zrachya
              Votes:
              1 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: