[CASSANDRA-16418] Unsafe to run nodetool cleanup during bootstrap or decommission

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.8, 4.1.1, 5.0-alpha1, 5.0
• Component/s: Consistency/Bootstrap and Decommission
• Bug Category: Correctness
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/8bb9c72f582de6bcc39522ba9ade91fd5bc22f67

Test and Documentation Plan

dtest, CHANGES.txt

• Assignee: Lindsey Zurovchak
• Reporter: James Baker

Authors

Lindsey Zurovchak

Reviewers

Paulo Motta, Stefan Miklosovic

• Created: 03/Feb/21 09:15
• Updated: 06/Dec/23 15:37
• Resolved: 23/Jan/23 22:45

Description

What we expected: Running a cleanup is a safe operation; the result of running a query after a cleanup should be the same as the result of running a query before a cleanup.

What actually happened: We ran a cleanup during a decommission. All the streamed data was silently deleted, the bootstrap did not fail, the cluster's data after the decommission was very different to the state before.

Why: Cleanups do not take into account pending ranges and so the cleanup thought that all the data that had just been streamed was redundant and so deleted it. We think that this is symmetric with bootstraps, though have not verified.

Not sure if this is technically a bug but it was very surprising (and seemingly undocumented) behaviour.

 

Issue Links

• Blocked: CASSANDRA-18824 Backport CASSANDRA-16418: Cleanup behaviour during node decommission caused missing replica
• links to: GitHub Pull Request #2921