[ZOOKEEPER-2574] PurgeTxnLog can inadvertently delete required txn log files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.7, 3.4.8, 3.5.0, 3.5.1, 3.5.2
• Fix Version/s: 3.4.10, 3.5.3, 3.6.0
• Component/s: server

Environment

Zookeeper 3.4.8, standalone, and 3-server quorum

• Assignee: Abhishek Rai
• Reporter: Abhishek Rai
• Created: 09/Sep/16 20:20
• Updated: 15/Nov/17 21:29
• Resolved: 23/Jan/17 01:31

Description

As part of the fix for ZOOKEEPER-1797, the call to FileTxnSnapLog.getSnapshotLogs() was removed from PurgeTxnLog.java. As a result, some old-looking but required txn log files can be deleted, resulting in data corruption or loss.

For example, consider the following:

1. Configuration:
autopurge.snapRetainCount=3

2. Following files exist:
log.100 spans transactions from zxid=100 till zxid=140 (inclusive)
snapshot.110 - snapshot as of zxid=110
snapshot.120 - snapshot as of zxid=120
snapshot.130 - snapshot as of zxid=130

Above scenario is possible when snapshotting has happened multiple times but without accompanying log rollover, which is possible if the server was running as a learner.

3. PurgeTxnLog retains all snapshots but deletes log.100 because its zxid is older than the zxid of the oldest snapshot (110). This results in loss of transactions in the range 131-140.

Before the fix for ZOOKEEPER-1797, this was avoided by the call to FileTxnSnapLog.getSnapshotLogs() which finds and retains the newest txn log file with starting zxid < oldest retained snapshot's highest zxid.

Issue Links

• duplicates: ZOOKEEPER-2420 Autopurge deletes log file prior to oldest retained snapshot even though restore may need it
• is broken by: ZOOKEEPER-1797 PurgeTxnLog may delete data logs during roll
• is related to: ZOOKEEPER-2671 Fix compilation error in branch-3.4
• links to: GitHub Pull Request #111