Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9393

Region Server fails to properly close socket resulting in many CLOSE_WAIT to Data Nodes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
    • 1.4.0, 1.3.2, 1.1.12, 2.0.0, 1.2.7
    • None
    • None

    • Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions

    • Reviewed
    • Hide
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.
      Show
      To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions.

    Description

      HBase dose not close a dead connection with the datanode.
      This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port.

      The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT

      [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
      13156
      [root@hd2-region3 ~]# ps -ef |grep 21592
      root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
      hbase 21592 1 17 Aug29 ? 03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...

      Attachments

        1. HBASE-9393.patch
          8 kB
          Ashish Singhi
        2. HBASE-9393.v1.patch
          4 kB
          Ashish Singhi
        3. HBASE-9393.v10.patch
          10 kB
          Ashish Singhi
        4. HBASE-9393.v11.patch
          10 kB
          Ashish Singhi
        5. HBASE-9393.v12.patch
          9 kB
          Ashish Singhi
        6. HBASE-9393.v13.patch
          10 kB
          Ashish Singhi
        7. HBASE-9393.v14.patch
          10 kB
          Ashish Singhi
        8. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        9. HBASE-9393.v15.patch
          10 kB
          Ashish Singhi
        10. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        11. HBASE-9393.v16.patch
          10 kB
          Ashish Singhi
        12. HBASE-9393.v17.patch
          11 kB
          Ashish Singhi
        13. HBASE-9393.v18.patch
          12 kB
          Ashish Singhi
        14. HBASE-9393.v2.patch
          5 kB
          Ashish Singhi
        15. HBASE-9393.v3.patch
          6 kB
          Ashish Singhi
        16. HBASE-9393.v4.patch
          6 kB
          Ashish Singhi
        17. HBASE-9393.v5.patch
          6 kB
          Michael Stack
        18. HBASE-9393.v5.patch
          6 kB
          Michael Stack
        19. HBASE-9393.v5.patch
          6 kB
          Ashish Singhi
        20. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        21. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        22. HBASE-9393.v6.patch
          7 kB
          Ashish Singhi
        23. HBASE-9393.v7.patch
          8 kB
          Ashish Singhi
        24. HBASE-9393.v8.patch
          9 kB
          Ashish Singhi
        25. HBASE-9393.v9.patch
          9 kB
          Ashish Singhi
        26. HBASE-9393-branch-1.patch
          14 kB
          Ashish Singhi

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-14747 S3AInputStream to implement CanUnbuffer

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14748 Wasb input streams to implement CanUnbuffer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-18212 In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-14765 AdlFsInputStream should implement unbuffer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12724 Let BufferedFSInputStream implement CanUnbuffer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12805 Annotate CanUnbuffer with @InterfaceAudience.Public

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HBASE-24155 When running the tests, a tremendous number of connections are put into TIME_WAIT.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              ashish singhi Ashish Singhi
              aviz Avi Zrachya
              Votes:
              1 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: