[CASSANDRA-16619] Loss of commit log data possible after sstable ingest

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.25, 3.11.11, 4.0-rc2, 4.0
• Component/s: Local/Commit Log
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 0.3

Source Control Link

https://github.com/apache/cassandra/commit/d84c6e98106e7b0c205f019ee24d416d0bb65f37

Test and Documentation Plan

run regression tests

• Assignee: Jacek Lewandowski
• Reporter: Jacek Lewandowski

Authors

Jacek Lewandowski

Reviewers

Benjamin Lerer, Branimir Lambov

• Created: 20/Apr/21 10:17
• Updated: 26/Apr/22 19:47
• Resolved: 17/May/21 08:47

Description

SSTable metadata contains commit log positions of the sstable. These positions are used to filter out mutations from the commit log on restart and only make sense for the node on which the data was flushed.

If an SSTable is moved between nodes they may cover regions that the receiving node has not yet flushed, and result in valid data being lost should these sections of the commit log need to be replayed.

Solution:
The chosen solution introduces a new sstable metadata (StatsMetadata) - originatingHostId (UUID), which is the local host id of the node on which the sstable was created, or null if not known. Commit log intervals from an sstable are taken into account during Commit Log replay only when the originatingHostId of the sstable matches the local node's hostId.

For new sstables the originatingHostId is set according to StorageService's local hostId.
For compacted sstables the originatingHostId set according to StorageService's local hostId, and only commit log intervals from local sstables is preserved in the resulting sstable.

discovered by jakubzytka

Issue Links

• causes: CASSANDRA-17042 Avoid sstable corruption when upgrading from 4.0-rc1 to 4.0.0