[ZOOKEEPER-2310] Snapshot files must be synced to prevent inconsistency or data loss

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.6
• Fix Version/s: 3.6.0
• Component/s: server
• Assignee: Abhishek Rai
• Reporter: Abhishek Rai
• Created: 01/Nov/15 06:26
• Updated: 11/Feb/21 20:37
• Resolved: 11/Feb/21 20:37

Description

Today, Zookeeper server syncs transaction log files to disk by default, but does not sync snapshot files. Consequently, an untimely crash may result in a lost or incomplete snapshot file. During recovery, if the server finds a valid older snapshot file, it will load it and replay subsequent log(s), skipping the incomplete snapshot file. It's possible that the skipped file had some transactions which are not present in the replayed transaction logs. Since quorum synchronization is based on last transaction ID of each server, this will never get noticed, resulting in inconsistency between servers and possible data loss.

Following sequence of events describes a sample scenario where this can happen:

1. Server F is a follower in a Zookeeper ensemble.
2. F's most recent valid snapshot file is named "snapshot.10" containing state up to zxid = 10. F is currently writing to the transaction log file "log.11", with the most recent zxid = 20.
3. Fresh round of election.
4. F receives a few new transactions 21 to 30 from new leader L as the "diff". Current server behavior is to dump current state plus diff to a new snapshot file, "snapshot.30".
5. F finalizes the snapshot file, but file contents are still buffered in OS caches. Zookeeper does not sync snapshot file contents to disk.
6. F receives a new transaction 31 from the leader, which it appends to the existing transaction log file, "log.11" and syncs the file to disk.
7. Server machine crashes or is cold rebooted.
8. After recovery, snapshot file "snapshot.30" may not exist or may be empty. See below for why that may happen.
9. In either case, F looks for the last finalized snapshot file, finds and loads "snapshot.10". It then replays transactions from "log.11". Ultimately, its last seen zxid will be 31, but it would not have replayed transactions 21 to 30 received via the "diff" from the leader.
10. Clients which are connected to F may see different data than clients connected to other members of the ensemble, violating single system image invariant. Also, if F were to become a leader at some point, it could use its state to seed other servers, and they all could lose the writes in the missing interval above.

Notes:

• Reason why snapshot file may be missing or incomplete:
  • Zookeeper does not sync the data directory after creating a snapshot file. Even if a newly created file is synced to disk, if the corresponding directory entry is not, then the file will not be visible in the namespace.
  • Zookeeper does not sync snapshot files. So, they may be empty or incomplete during recovery from an untimely crash.
• In step (6) above, the server could also have written the new transaction 31 to a new log file, "log.31". The final outcome would still be the same.

We are able to deterministically reproduce this problem using the following steps:

1. Create a new Zookeeper ensemble on 3 hosts: A, B, and C.
2. Ensured each server has at least one snapshot file in its data dir.
3. Stop Zookeeper process on server A.
4. Slow down disk syncs on server A (see example script below). This ensures that snapshot files written by Zookeeper don't make it to disk spontaneously. Log files will be written to disk as Zookeeper explicitly issues a sync call on such files.
5. Connect to server B and create a new znode /test1.
6. Start Zookeeper process on A, wait for it to write a new snapshot to its datadir. This snapshot would contain /test1 but it won’t be synced to disk yet.
7. Connect to A and verify that /test1 is visible.
8. Connect to B and create another znode /test2. This will cause A’s transaction log to grow further to receive /test2.
9. Cold reboot A.
10. A’s last snapshot is a zero-sized file or is missing altogether since it did not get synced to disk before reboot. We have seen both in different runs.
11. Connect to A and verify that /test1 does not exist. It exists on B and C.

Slowing down disk syncs:

echo 360000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs
echo 360000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo 99 | sudo tee /proc/sys/vm/dirty_background_ratio
echo 99 | sudo tee /proc/sys/vm/dirty_ratio

Issue Links

• is fixed by: ZOOKEEPER-2872 Interrupted snapshot sync causes data loss