[CURATOR-220] LOST state is sometimes not reported to ConnectionStateListener

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.2, 2.8.0
• Fix Version/s: awaiting-response
• Component/s: Framework
• Assignee: Unassigned
• Reporter: Akos Gyimesi
• Created: 15/May/15 20:11
• Updated: 16/Oct/15 19:34

Description

I used iptables to drop all outgoing packets to ZooKeeper, and logged the connection state changes. Most of the time I got only CONNECTED->SUSPENDED->RECONNECTED states, even though the ZooKeeper session expired and the ephemeral nodes of the session disappeared. (I expected CONNECTED->SUSPENDED->LOST->RECONNECTED)

According to CURATOR-185 this may not be a bug because Curator connection states are not in 1-to-1 relation with ZooKeeper connection events. However, it causes problems in the LeaderSelector recipe (and possibly in others): without the LOST event the leader will never know if it really lose leadership or not. If it does not resign leadership at the SUSPENDED event (which is recommended, but not required according to the docs) the session will be in leader state forever.

See the following gist: https://gist.github.com/gyim/caea1b73cc8fa8f6997b
The code tests the LeaderSelector recipe, but the behavior is the same with a simple ConnectionStateListener.

Tested with Curator 2.4.2 and 2.8.0, it probably affects several other versions as well.