[CASSANDRA-14559] Check for endpoint collision with hibernating nodes

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.22, 3.11.8, 4.0-beta2, 4.0
• Component/s: Consistency/Bootstrap and Decommission
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 1.0.0

Source Control Link

https://github.com/apache/cassandra/commit/c94ececec0fcd87459858370396d6cd586853787

• Assignee: Stefan Miklosovic
• Reporter: Vincent White

Authors

Stefan Miklosovic

Reviewers

Brandon Williams

• Created: 06/Jul/18 04:05
• Updated: 08/May/21 00:11
• Resolved: 04/Aug/20 22:48

Description

I ran across an edge case when replacing a node with the same address. This issue results in the node(and its tokens) being unsafely removed from gossip.

Steps to replicate:

1. Create 3 node cluster.
2. Stop a node
3. Replace the stopped node with a node using the same address using the replace_address flag
4. Stop the node before it finishes bootstrapping
5. Remove the replace_address flag and restart the node to resume bootstrapping (if the data dir is also cleared at this point the node will also generate new tokens when it starts)
6. Stop the node before it finishes bootstrapping again
7. 30 Seconds later the node will be removed from gossip because it now matches the check for a FatClient

I think this is only an issue when replacing a node with the same address because other replacements now use STATUS_BOOTSTRAPPING_REPLACE and leave the dead node unchanged.

I believe the simplest fix for this is to add a check that prevents a non-bootstrapped node (without the replaces_address flag) starting if there is a gossip entry for the same address in the hibernate state.

3.11 PoC

 

Issue Links

• is related to: CASSANDRA-16662 Move CASSANDRA-14559s bootstrap_test.py::TestBootstrap::test_node_cannot_join_as_hibernating_node_without_replace_address into a jvm-dtest