[CASSANDRA-9878] Rolling Updates 2.0 to 2.1 "unable to gossip"

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Severity: Normal
• Assignee: Unassigned
• Reporter: Axel Kämpfe
• Created: 23/Jul/15 11:22
• Updated: 16/Apr/19 09:31
• Resolved: 23/Jul/15 14:33

Description

Hi there,

we are currently testing an upgrade of our servers from Cassandra 2.0.16 to 2.1.8 on the EC2 service of Amazon

Usually, we launch a new server which gets the newest version and then joins the existing ring, bootstrapping, and after some time we kill one of the old nodes.

But with the upgrade to 2.1 the new server will not join the existing ring.

INFO  [HANDSHAKE-/10.] 2015-07-23 11:06:14,997 OutboundTcpConnection.java:485 - Handshaking version with /10.xx.yy.zz
INFO  [HANDSHAKE-/10.] 2015-07-23 11:06:14,999 OutboundTcpConnection.java:485 - Handshaking version with /10.yy.zz.aa
INFO  [HANDSHAKE-/10.] 2015-07-23 11:06:15,000 OutboundTcpConnection.java:485 - Handshaking version with /10.aa.bb.cc
ERROR [main] 2015-07-23 11:06:46,016 CassandraDaemon.java:541 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1307) ~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:533) ~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:777) ~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:714) ~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:605) ~[apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8]
WARN  [StorageServiceShutdownHook] 2015-07-23 11:06:46,023 Gossiper.java:1418 - No local state or state is in silent shutdown, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2015-07-23 11:06:46,023 MessagingService.java:708 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/10.] 2015-07-23 11:06:46,045 MessagingService.java:958 - MessagingService has terminated the accept() thread

our config uses the "RandomPartitioner", "SimpleSnitch" and the internal nodes ip for communication.

when i use the same config but ONLY 2.1.x Servers, everthing works perfectly, but as soon as we start to "mix in" a new version into an "old" ring, the new servers will not "gossip"... ( so it cannot be a firewall issue, or such)

If you need any more information from my side, please let me know.

Issue Links

• duplicates: CASSANDRA-8072 Exception during startup: Unable to gossip with any seeds