Zookeeper always tries to accept new connections, throwing an exception if out of file descriptors. An improvement would be denying new client connections when close to the limit.
Additionally, file-descriptor limits+usage should be exported to the monitoring four-letter word, should that get implemented (see
A Zookeeper ensemble I administer recently suffered an outage when one node was restarted with the low system-default ulimit of 1024 file descriptors and later ran out. File descriptor usage+max are already being monitored by the following MBeans:
They're described (rather tersely) at:
This feature request is for the following:
(a) Stop accepting new connections when OpenFileDescriptorCount is close to MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be denied, logged to disk at debug level, and increment a ``ConnectionDeniedCount`` MBean counter.
(b) Begin accepting new connections when usage drops below some configurable threshold, defaulting to 90% of FD usage, basically the high/low watermark model.
(c) Update the administrators guide with a comment about using an appropriate FD limit.
(d) Extra credit: if
ZOOKEEPER-744 is implemented export statistics for:
zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not all zk's have the same max FD value