[CASSANDRA-11892] Can not replace a dead host

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Information Provided
• Severity: Normal
• Since Version: 2.1.14
• Assignee: Unassigned
• Reporter: Dikang Gu
• Created: 25/May/16 21:52
• Updated: 16/Apr/19 09:30
• Resolved: 19/Nov/18 02:27

Description

I got some errors when trying to replace a dead host.

2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup
2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61841       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token 100284002935427428580945058996711341062 which does not exist!

the status of the node is DN:

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address                          Load       Tokens  Owns    Host ID                               Rack
DN  2401:db00:2050:4196:face:0:13:0  809.83 GB  256     ?       null                                  ash5-04-pp

I add some logging and find something like this:

2016-05-25_20:58:33.44305 INFO  20:58:33 [main]: Gathering node replacement information for /2401:db00:2050:4196:face:0:13:0
2016-05-25_20:58:34.36966 INFO  20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12167 INFO  20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12248 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state STATUS
2016-05-25_20:58:41.12250 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 movename hibernate
2016-05-25_20:58:41.12252 INFO  20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state LOAD

I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue, does it cause the problem?

Is it safe to apply the patch to fix it?