[ZOOKEEPER-3145] Potential watch missing issue due to stale pzxid when replaying CloseSession txn with fuzzy snapshot

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.5.4, 3.6.0, 3.4.13
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Fangmin Lv
• Reporter: Fangmin Lv
• Created: 11/Sep/18 23:47
• Updated: 15/Oct/24 03:27
• Resolved: 11/Sep/19 08:27

Description

This is another issue I found recently, we haven't seen this problem on prod (or maybe we don't notice).

 
Currently, the CloseSession is not idempotent, executing the CloseSession twice won't get the same result.
 
The problem is that closeSession will only check what's the ephemeral nodes associated with that session bases on current states. Nodes deleted during taking fuzzy snapshot won't be deleted again when replay the txn.
 
This looks fine, since it's already gone, but there is problem with the pzxid of the parent node. Snapshot is taken fuzzily, so it's possible that the parent had been serialized while the nodes are being deleted when executing the closeSession Txn. The pzxid will not be updated in the snapshot when replaying the closeSession txn, because doesn't know what's the paths being deleted, so it won't patch the pzxid like what we did in the deleteNode ZOOKEEPER-3125.
 
The inconsistent pzxid will lead to potential watch notification missing when client reconnect with setWatches because of the staleness. 
 
This JIRA is going to fix those issues by adding the CloseSessionTxn, it will record all those nodes being deleted in that CloseSession txn, so that we know which nodes to update when replaying the txn.

Issue Links

• causes: ZOOKEEPER-4306 CloseSessionTxn contains too many ephemal nodes cause cluster crash
• is related to: ZOOKEEPER-4874 Avoid using fuzzy snapshots.
• links to: GitHub Pull Request #622
• links to: GitHub Pull Request #622