[ZOOKEEPER-2325] Data inconsistency if all snapshots empty or missing

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.6
• Fix Version/s: 3.5.4, 3.6.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Andrew Grasso
• Reporter: Andrew Grasso
• Created: 18/Nov/15 22:21
• Updated: 21/Aug/19 16:59
• Resolved: 26/Oct/18 10:04

Description

When loading state from snapshots on startup, FileTxnSnapLog.java ignores the result of FileSnap.deserialize, which is -1L if no valid snapshots are found. Recovery proceeds with dt.lastProcessed == 0, its initial value.

The result is that Zookeeper will process the transaction logs and then begin serving requests with a different state than the rest of the ensemble.

To reproduce:
In a healthy zookeeper cluster of size >= 3, shut down one node.
Either delete all snapshots for this node or change all to be empty files.
Restart the node.

We believe this can happen organically if a node runs out of disk space.

Issue Links

• causes: ZOOKEEPER-3513 Zookeeper upgrade fails due to missing snapshots
• requires: ZOOKEEPER-3056 Fails to load database with missing snapshot file but valid transaction log file
• links to: GitHub Pull Request #117
• links to: GitHub Pull Request #144
• links to: GitHub Pull Request #144