[CASSANDRA-16213] Cannot replace_address /X because it doesn't exist in gossip

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta4, 4.0
• Component/s: Cluster/Gossip, Cluster/Membership
• Bug Category: Availability - Unavailable
• Severity: Critical
• Complexity: Challenging
• Discovered By: User Report
• Platform: All
• Since Version: 2.0.0

Source Control Link

https://github.com/apache/cassandra/commit/d61710d5b2623f0c1ebf4dbcbfaaf6c8d3a3ab91

Test and Documentation Plan

tests added

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell

Reviewers

Brandon Williams, Jon Meredith, Paulo Motta, Sam Tunnicliffe

• Created: 15/Oct/20 19:11
• Updated: 03/Jan/21 16:58
• Resolved: 15/Dec/20 22:55

Description

We see this exception around nodes crashing and trying to do a host replacement; this error appears to be correlated around multiple node failures.

A simplified case to trigger this is the following

*) Have a N node cluster
*) Shutdown all N nodes
*) Bring up N-1 nodes (at least 1 seed, else replace seed)
*) Host replace the N-1th node -> this will fail with the above

The reason this happens is that the N-1th node isn’t gossiping anymore, and the existing nodes do not have its details in gossip (but have the details in the peers table), so the host replacement fails as the node isn’t known in gossip.

This affects all versions (tested 3.0 and trunk, assume 2.2 as well)

Issue Links

• links to: Circle CI: trunk
• links to: GH PR: trunk