[CASSANDRA-11886] Streaming will miss sections for early opened sstables during compaction

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1.15, 2.2.7, 3.0.8, 3.8
• Labels: correctness repair streaming
• Severity: Critical
• Assignee: Marcus Eriksson
• Reporter: Stefan Podkowinski

Authors

Marcus Eriksson

Reviewers

Benedict Elliott Smith

• Created: 24/May/16 15:48
• Updated: 16/Apr/19 09:30
• Resolved: 13/Jun/16 13:33

Description

Once validation compaction has been finished, all mismatching sstable sections for a token range will be used for streaming as return by StreamSession.getSSTableSectionsForRanges. Currently 2.1 will try to restrict the sstable candidates by checking if they can be found in CANONICAL_SSTABLES and will ignore them otherwise. At the same time IntervalTree in the DataTracker will be build based on replaced non-canonical sstables as well. In case of early opened sstables this becomes a problem, as the tree will be update with OpenReason.EARLY replacements that cannot be found in canonical. But whenever getSSTableSectionsForRanges will get a early instance from the view, it will fail to retrieve the corresponding canonical version from the map, as the different generation will cause a hashcode mismatch. Please find a test attached.

As a consequence not all sections for a range are streamed. In our case this has caused deleted data to reappear, as sections holding tombstones were left out due to this behavior.

Issue Links

• is broken by: CASSANDRA-6916 Preemptive opening of compaction result
• is related to: CASSANDRA-9462 ViewTest.sstableInBounds is failing
• is related to: CASSANDRA-9700 StreamSession selects canonical sstables, but uses the non-canonical instance