[CASSANDRA-7292] Can't seed new node into ring with (public) ip of an old node

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.15, 2.1.5
• Labels: bootstrap gossip

Environment

Cassandra 2.0.7, Ec2MultiRegionSnitch

• Severity: Normal
• Assignee: Brandon Williams Assign to me
• Reporter: Juho Mäkinen

Authors

Brandon Williams

Reviewers

John Alberts

• Created: 23/May/14 08:01
• Updated: 16/Apr/19 09:31
• Resolved: 17/Apr/15 22:39

Description

This bug prevents node to return with bootstrap into the cluster with its old ip.

Scenario: five node ec2 cluster spread into three AZ, all in one region. I'm using Ec2MultiRegionSnitch. Nodes are reported with their public ips (as Ec2MultiRegionSnitch requires)

I simulated a loss of one node by terminating one instance. nodetool status reported correctly that node was down. Then I launched new instance with the old public ip (i'm using elastic ips) with "Dcassandra.replace_address=IP_ADDRESS" but the new node can't join the cluster:

INFO 07:20:43,424 Gathering node replacement information for /54.86.191.30
INFO 07:20:43,428 Starting Messaging Service on port 9043
INFO 07:20:43,489 Handshaking version with /54.86.171.10
INFO 07:20:43,491 Handshaking version with /54.86.187.245
(some delay)
ERROR 07:21:14,445 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193)
at org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:419)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:650)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569)

It does not help if I remove the "Dcassandra.replace_address=IP_ADDRESS" system property.

Also it does not help to remove the node with "nodetool removenode" with or without the cassandra.replace_address property.

I think this is because the node information is preserved in the gossip info as seen this output of "nodetool gossipinfo"
/54.86.191.30
INTERNAL_IP:172.16.1.231
DC:us-east
REMOVAL_COORDINATOR:REMOVER,d581309a-8610-40d4-ba30-cb250eda22a8
STATUS:removed,19311925-46b5-4fe4-928a-321e8adb731d,1401089960664
HOST_ID:19311925-46b5-4fe4-928a-321e8adb731d
RPC_ADDRESS:0.0.0.0
NET_VERSION:7
SCHEMA:226f9315-b4b2-32c1-bfe1-f4bb49fccfd5
RACK:1b
LOAD:7.075290515E9
SEVERITY:0.0
RELEASE_VERSION:2.0.7

Issue Links

• duplicates: CASSANDRA-8072 Exception during startup: Unable to gossip with any seeds