[CASSANDRA-17042] Avoid sstable corruption when upgrading from 4.0-rc1 to 4.0.0

• Type: Bug
• Status: Review In Progress
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.0.x, 5.x
• Component/s: Consistency/Streaming, Local/SSTable
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Critical
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All

Test and Documentation Plan

jvm dtest, cci run

• Assignee: Marcus Eriksson
• Reporter: Marcus Eriksson

Authors

Marcus Eriksson

Reviewers

Caleb Rackliffe

• Created: 14/Oct/21 07:19
• Updated: 07/Mar/23 10:53

Description

4.0-rc2 introduced a new sstable version (nb), adding a field in StatsMetadata. When zero copy streaming a file from 4.0-rc2+ to 4.0-rc1, the rc1 node will write it as an nb file, mutate the metadata to clear repairedAt/level information from StatsMetadata and rewrite the file, keeping the same file name. Problem is that since rc1 doesn't know about the new field in StatsMetadata, it will write the file without the new StatsMetadata field, but with an nb filename. Then, when upgrading this node to rc2+ we will try to read it as a nb file, but will get EOFException since it doesn't contain the new field.

We should make sure that we never do zero-copy streaming to nodes that don't know about the file version.

The corrupt sstables can be fixed by simply renaming all components of it to na from nb

Issue Links

• is caused by: CASSANDRA-16619 Loss of commit log data possible after sstable ingest