[CASSANDRA-16873] Tolerate missing DNS entry when completing a host replacement

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.1, 4.1-alpha1, 4.1
• Component/s: Cluster/Membership
• Change Category: Operability
• Complexity: Normal
• Platform: All

Source Control Link

https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=f59411f1c985043850154971e9c4066013f355cb

Test and Documentation Plan

New unit testing, and no new documentation required.

• Assignee: Josh McKenzie
• Reporter: Josh McKenzie

Authors

Chris Lohfink, Josh McKenzie

Reviewers

Brandon Williams

• Created: 20/Aug/21 16:57
• Updated: 27/May/22 19:25
• Resolved: 27/Aug/21 19:55

Description

In one of our deployments, after a host replacement a subset of nodes still saw the nodes as JOINING despite the rest of the cluster seeing it as NORMAL with a failure to gossip. This was traced to a DNS lookup failure on the nodes during an interim state leading to an exception being thrown and gossip state never transitioning.

Rather than implicitly requiring operators to bounce the node by throwing an exception, we should instead suppress the exception when checking if a node is replacing the same host address and ID if we get an UnknownHostException.