[ZOOKEEPER-4882] Data loss after restarting an node experienced temporary disk error and rejoin

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.8.4, 3.9.3
• Component/s: server
• Assignee: Unassigned
• Reporter: Kezhu Wang
• Created: 27/Oct/24 03:08
• Updated: 28/Oct/24 04:17

Description

The cause is multifold:
1. Leader will commit a proposal once quorum acked.
2. Proposal is able to be committed in node's memory even if it has not
been written to that node's disk.
3. In case of disk error, the txn log could lag behind memory database.

The above applies to both leader and follower. I have not verified leader branch, let's consider only follower for now.

f4. A follower experienced temporary disk error will have hole in txn log
after re-join.
f5. Restarted follower will lose the data. Worse, it is able to win
election and propagate data loss to whole cluster.

I authored commits in my repo to expose this.

https://github.com/kezhuw/zookeeper/commits/data-loss-temporary-sync-disk-error/

Issue Links

• blocks: ZOOKEEPER-4766 Ensure leader election time does not unnecessarily scale with tree size due to snapshotting
• relates to: ZOOKEEPER-4766 Ensure leader election time does not unnecessarily scale with tree size due to snapshotting