[CASSANDRA-16718] Changing listen_address with prefer_local may lead to issues

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.10, 4.1.2, 5.0-alpha1, 5.0
• Component/s: Local/Config
• Bug Category: Availability
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/b791644fda91b343a679bb0c2c1e33e594524636

Test and Documentation Plan

test included, CI

• Assignee: Brandon Williams
• Reporter: Jan Karlsson

Authors

Brandon Williams

Reviewers

Berenguer Blasi

• Created: 07/Jun/21 12:49
• Updated: 12/Sep/23 13:02
• Resolved: 16/May/23 14:04

Description

Many container based solution function by assigning new listen_addresses when nodes are stopped. Changing the listen_address is usually as simple as turning off the node and changing the yaml file. 

However, if prefer_local is enabled, I observed that nodes were unable to join the cluster and fail with 'Unable to gossip with any seeds'.

Trace shows that the changing node will try to communicate with the existing node but the response is never received. I assume it is because the existing node attempts to communicate with the local address during the shadow round.

 

Issue Links

• causes: CASSANDRA-18560 Incorrect IP used for gossip across DCs with prefer_local=true