[ZOOKEEPER-1049] Session expire/close flooding renders heartbeats to delay significantly

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.3.4, 3.4.0
• Component/s: server

Environment

CentOS 5.3, three node ZK ensemble

• Assignee: Chang Song
• Reporter: Chang Song
• Created: 16/Apr/11 03:42
• Updated: 23/Nov/11 19:22
• Resolved: 03/May/11 21:30

Description

Let's say we have 100 clients (group A) already connected to three-node ZK ensemble with session timeout of 15 second. And we have 1000 clients (group B) already connected to the same ZK ensemble, all watching several nodes (with 15 second session timeout)

Consider a case in which All clients in group B suddenly hung or deadlocked (JVM OOME) all at the same time. 15 seconds later, all sessions in group B gets expired, creating session closing stampede. Depending on the number of this clients in group B, all request/response ZK ensemble should process get delayed up to 8 seconds (1000 clients we have tested).

This delay causes some clients in group A their sessions expired due to delay in getting heartbeat response. This causes normal servers to drop out of clusters. This is a serious problem in our installation, since some of our services running batch servers or CI servers creating the same scenario as above almost everyday.

I am attaching a graph showing ping response time delay.

I think ordering of creating/closing sessions and ping exchange isn't important (quorum state machine). at least ping request / response should be handle independently (different queue and different thread) to keep realtime-ness of ping.

As a workaround, we are raising session timeout to 50 seconds.
But this causes max. failover of cluster to significantly increased, thus initial QoS we promised cannot be met.

Issue Links

• relates to: ZOOKEEPER-1238 when the linger time was changed for NIO the patch missed Netty