[ZOOKEEPER-2926] Data inconsistency issue due to the flaw in the session management

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.5.3, 3.6.0
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Fangmin Lv
• Reporter: Fangmin Lv
• Created: 27/Oct/17 23:23
• Updated: 08/Aug/18 07:22
• Resolved: 08/Aug/18 04:21

Description

The local session upgrading feature will upgrade the session locally before receving a quorum commit of creating global session. It's possible that the server shutdown before the creating session request being sent to leader, if we retained the ZKDatabase or there is Snapshot happened just before shutdown, then only this server will have the global session.

If that server didn't become leader, then it will have more global sessions than others, and those global sessions won't expire as the leader doesn't know it's existence. If the server became leader, it will accept the client renew session request and the client is allowed to create ephemeral nodes, which means other servers only have ephemeral nodes but not that global session. If there is follower going to have SNAP sync with it, then it will also have the global session. If the server without that global session becomes new leader, it will check and delete those dangling ephemeral node before serving traffic. These could introduce the issues that the ephemeral node being exist on some servers but not others.

There is dangling global session issue even without local session feature, because on leader it will update the ZKDatabase when processing ConnectionRequest and in the PrepRequestProcessor before it's quorum committed, which also has this risk.

Issue Links

• links to: GitHub Pull Request #447