Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-759

Stop accepting connections when close to file descriptor limit

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.6.0, 3.5.5
    • Component/s: server
    • Labels:
      None

      Description

      Zookeeper always tries to accept new connections, throwing an exception if out of file descriptors. An improvement would be denying new client connections when close to the limit.

      Additionally, file-descriptor limits+usage should be exported to the monitoring four-letter word, should that get implemented (see ZOOKEEPER-744).

      DETAILS

      A Zookeeper ensemble I administer recently suffered an outage when one node was restarted with the low system-default ulimit of 1024 file descriptors and later ran out. File descriptor usage+max are already being monitored by the following MBeans:

      • java.lang.OperatingSystem.MaxFileDescriptorCount
      • java.lang.OperatingSystem.OpenFileDescriptorCount

      They're described (rather tersely) at:

      http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html

      This feature request is for the following:

      (a) Stop accepting new connections when OpenFileDescriptorCount is close to MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be denied, logged to disk at debug level, and increment a ``ConnectionDeniedCount`` MBean counter.

      (b) Begin accepting new connections when usage drops below some configurable threshold, defaulting to 90% of FD usage, basically the high/low watermark model.

      (c) Update the administrators guide with a comment about using an appropriate FD limit.

      (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for:

      zookeeper_open_file_descriptor_count
      zookeeper_max_file_descriptor_count
      zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not all zk's have the same max FD value

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              traviscrawford Travis Crawford
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: