[CASSANDRA-8072] Exception during startup: Unable to gossip with any seeds

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.13, 2.2.5, 3.0.3, 3.3
• Component/s: Local/Startup and Shutdown
• Severity: Normal
• Assignee: Stefania Alborghetti Assign to me
• Reporter: Ryan Springer

Authors

Stefania Alborghetti

Reviewers

Sam Tunnicliffe

• Tester: Russ Hatch
• Created: 07/Oct/14 16:05
• Updated: 16/Apr/19 09:31
• Resolved: 08/Jan/16 16:13

Description

When Opscenter 4.1.4 or 5.0.1 tries to provision a 2-node DSC 2.0.10 cluster in either ec2 or locally, an error occurs sometimes with one of the nodes refusing to start C*. The error in the /var/log/cassandra/system.log is:

ERROR [main] 2014-10-06 15:54:52,292 CassandraDaemon.java (line 513) Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:444)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:655)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:609)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:502)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
INFO [StorageServiceShutdownHook] 2014-10-06 15:54:52,326 Gossiper.java (line 1279) Announcing shutdown
INFO [StorageServiceShutdownHook] 2014-10-06 15:54:54,326 MessagingService.java (line 701) Waiting for messaging service to quiesce
INFO [ACCEPT-localhost/127.0.0.1] 2014-10-06 15:54:54,327 MessagingService.java (line 941) MessagingService has terminated the accept() thread

This errors does not always occur when provisioning a 2-node cluster, but probably around half of the time on only one of the nodes. I haven't been able to reproduce this error with DSC 2.0.9, and there have been no code or definition file changes in Opscenter.

I can reproduce locally with the above steps.  I'm happy to test any proposed fixes since I'm the only person able to reproduce reliably so far.

Issue Links

• is depended upon by: CASSANDRA-8175 Extend existing endpoint check to also check when restarting normally
• is duplicated by: CASSANDRA-7292 Can't seed new node into ring with (public) ip of an old node
• is duplicated by: CASSANDRA-9878 Rolling Updates 2.0 to 2.1 "unable to gossip"
• relates to: CASSANDRA-8422 cassandra won't start up due to "Unable to gossip with any seeds" on the decommissioned node