[CASSANDRA-7350] Decommissioning nodes borks the seed node - can't add additional nodes

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Cannot Reproduce
• Fix Version/s: 2.0.9
• Labels: qa-resolved

Environment

Ubuntu using the auto-clustering AMI

• Severity: Low
• Assignee: Shawn Kumar
• Reporter: Steven Lowenthal

Authors

Shawn Kumar

• Tester: Shawn Kumar
• Created: 03/Jun/14 22:16
• Updated: 16/Apr/19 09:31
• Resolved: 19/Jun/14 22:43

Description

1) Launch a 4 node cluster - I used the auto-clustering AMI (you get nodes 0-3)
2) decommission that last 2 nodes (nodes , leaving a 2 node cluster)
3) wipe the data directories from node 2
4) bootstrap node2 - it won't join "unable to gossip with any seeds".

If you bootstrap the node a second time, it will join. However if you try to bootstrap node 3, it will also fail.

I discovered that bouncing the seed node fixes the problem. I think it cropped up in 2.0.7.

Error:
ERROR [main] 2014-06-03 21:52:46,649 CassandraDaemon.java (line 497) Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569)
ERROR [StorageServiceShutdownHook] 2014-06-03 21:52:46,741 CassandraDaemon.java (line 198) Exception in thread Thread[StorageServi