[CASSANDRA-17273] Lazy transaction log replica creation allows incorrect replica content divergence during compaction

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.27, 3.11.13, 4.0.4, 4.1-alpha1, 4.1
• Component/s: Consistency/Repair, Local/Compaction
• Bug Category: Availability - Process Crash
• Severity: Critical
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/85c202d8a8b037791db08c531f4f3c8336c82696

Test and Documentation Plan

new unit test, cci run

• Assignee: Marcus Eriksson
• Reporter: Caleb Rackliffe

Authors

Marcus Eriksson

Reviewers

Caleb Rackliffe

• Created: 20/Jan/22 22:49
• Updated: 27/May/22 19:24
• Resolved: 16/Feb/22 10:22

Description

Recently encountered this around compaction/anticompaction:

2022-01-13 10:18:24,325 ERROR [main] org.apache.cassandra.db.lifecycle.LogTransaction - Unexpected disk state: failed to read transaction log [mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log in .../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f, .../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f]
Files and contents follow:
.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
	ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
	REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
	REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
	COMMIT:[,0,0][2613697770]
.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/mf_txn_anticompactionafterrepair_2f826324-742c-11ec-b293-65cae21e111c.log
	ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350437-big,0,8][4051162457]
		***Does not match <ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]> in first replica file
	ADD:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350438-big,0,8][2380834168]
	REMOVE:[.../d2/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350435-big,1642049328006,8][2338829485]
	REMOVE:[.../d1/data/.../files-c351f12917af3a5cbc57791cdf178a1f/prod_p203-files-mf-350436-big,1642049366291,8][4248366924]
	COMMIT:[,0,0][2613697770]

We have two data directories and two transaction log files, but one is missing an ADD entry when the contents of the two log replicas should be identical. One scenario that can cause this is the following:

1. Start anticompaction on a single file, in directory /tmp/d0.

2. Call trackNew() with 2 new files, both in a single directory, but in directory /tmp/d1. This initializes the log file in /tmp/d1, but there is still no log file in /tmp/d0.

3. Anticompaction only writes to one of the files in /tmp/d1 (say all other keys were outside the repaired range).

4. When anticompaction is done, the empty writer is aborted and we call untrackNew(), which removes the added file from the registered log “records" (BUT NOT FROM DISK in /tmp/d1).

5. The REMOVE record is added. This references /tmp/d0. We lazily create the log file there by dumping all the records we have in memory to that file, which does not include the aborted SSTable above.

6. Now the log files contain:

/tmp/d1/logfile.log:
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-2-big,0,8][3268492367]
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
COMMIT:[,0,0][2613697770]
** /tmp/d0/logfile.log:
ADD:[/tmp/d1/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-3-big,0,8][2813724425]
REMOVE:[/tmp/d0/AntiCompactionTest/AntiCompactionTest-e4fdddf0746e11ecb73ad5a997381615/AntiCompactionTest-AntiCompactionTest-mf-1-big,1642078019000,8][2401235379]
COMMIT:[,0,0][2613697770]