[ZOOKEEPER-3607] Potential data inconsistency due to the inconsistency between ZKDatabase.committedLog and dataTree in Trunc sync.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.4.14
• Component/s: quorum
• Assignee: Unassigned
• Reporter: Jiafu Jiang
• Created: 06/Nov/19 07:33
• Updated: 13/Mar/20 06:44

Description

I will describe the problem by a detail example.

1. Suppose we have three zk servers: zk1, zk2, and zk3. zk1 and zk2 are online, zk3 is offline, zk1 is the leader.

2. In TRUNC sync, zk1 sends a TRUNC request to zk2, then sends the remaining proposals in the committedLog. When the follower zk2 receives the proposals, it applies them directly into the datatree, but not the committedLog.

3. After the data sync phase, zk1 may continue to send zk2 more committed proposals, and they will be applied to both the datatree and the committedLog of zk2.

4. Then zk1 fails, zk3 restarts successfully, zk2 becomes the leader.

5. The leader zk2 sends a TRUNC request to zk3, then the remaining proposals from the committedLog. But since some proposals, which are from the leader zk1 in TRUNC sync(as I describe above), are not in the committedLog, they will not be sent to zk3.

6. Now data inconsistency happens between zk2 and zk3, since some data may exist in zk2's datatree, but not zk3's datatree.