[CASSANDRA-17896] Failed replacements due to "Cannot replace token xxx which does not exist"

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Won't Fix
• Fix Version/s: 5.x
• Component/s: Consistency/Streaming
• Bug Category: Degradation - Other Exception
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All

Test and Documentation Plan

New distributed testing confirming functionality works as expected and doesn't disrupt existing functionality

• Assignee: Josh McKenzie
• Reporter: Josh McKenzie

Authors

Josh McKenzie

• Created: 14/Sep/22 18:59
• Updated: 07/Mar/23 10:53
• Resolved: 22/Sep/22 17:32

Description

Host replacements require host ID and token, however reporting of this info is typically blocked when the “status” field is missing. We should expose a system env (-D) param to advanced operators to have the ability to specify the replace_addresses_token to be used during host replacement in cases where Gossip gets into a bad state.

Given this is inherently an unsafe / high risk manual intervention, we want to avoid promoting this to being a config param however Gossip quite often contorts into problematic split-brain states like this and having the ability to manually override things helps unstick host replacements that go straddled. Should update NEWS.txt at the very least and consider documentation as well

This is an interim solution that'll be obviated by CEP-21, but is still useful on existing clusters.