Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-4733

Rest client does not close zookeeper sessions (leaking sessions for each GET or PUT)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 0.90.4
    • 0.90.4
    • REST
    • None

    • Fedora 13.

    • This fix works on our 60 node hbase 0.90.4 cluster running CDH3U2 hadoop.

    Description

      This will appear in the log once the zookeeper connection/session leaking will grow to 2000, zookeeper won't be able to accept any more connections causing REST RPC calls to fail, here is the log when the problem is in progress:

      2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
      2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
      2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
      2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
      2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
      2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
      2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.4 - max is 2000
      2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
      2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
      2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000
      2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
      2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.3 - max is 2000
      2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.3 - max is 2000
      2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.101.255.2 - max is 2000
      2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too many connections from /10.102.255.2 - max is 2000

      After the fix, the log looks much better, and we can observe zookeeper connections closing after every RPC call:

      2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:37225
      2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:37225
      2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f19f with negotiated timeout 180000 for client /10.101.255.2:37225
      2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:37225 which had sessionid 0x3352857cb1f19f
      2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.2:38089
      2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.2:38089
      2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client /10.101.255.2:38089
      2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.101.255.2:38089 which had sessionid 0x3352857cb1f1a0
      2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /10.101.255.3:34085
      2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client attempting to establish new session at /10.101.255.3:34085
      2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn: Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client /10.101.255.3:34085

      Attachments

        1. my.patch
          0.9 kB
          jack levin

        Activity

          People

            Unassigned Unassigned
            jacque74 jack levin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: