[CASSANDRA-15335] Node can corrupt gossip state and become unreplaceable

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.19, 3.11.5, 4.0-alpha2, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Correctness - Consistency
• Severity: Low
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/39a431d9d821cf6bcb54da7198b6a64e09b0f120

Test and Documentation Plan

circle

• Assignee: Blake Eggleston
• Reporter: Blake Eggleston

Authors

Blake Eggleston

Reviewers

Marcus Eriksson, Sam Tunnicliffe

• Created: 24/Sep/19 18:10
• Updated: 15/May/20 08:36
• Resolved: 07/Oct/19 18:50

Description

In StorageService#prepareToJoin, a starting node first sends out an endpoint state without any tokens. Later, in StorageService#finishJoiningRing it sends out an endpoint state with it’s tokens. If that node dies between these 2 events and cannot be restarted due to some unrecoverable error, the ring’s gossip state will be missing tokens for that node. This won’t cause any immediate data loss since TMD is populated from system.peers, but it will prevent a replacement node from associating that address with it’s tokens and replacing it. It could also cause data loss if other nodes are added to the ring and don’t see an owned token where there should be one.