[CASSANDRA-12172] Fail to bootstrap new node.

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Severity: Normal
• Since Version: 2.2.5
• Assignee: Unassigned
• Reporter: Dikang Gu
• Created: 12/Jul/16 05:22
• Updated: 16/Apr/19 09:30
• Resolved: 05/Jan/17 18:24

Description

When I try to bootstrap new node in the cluster, sometimes it failed because of following exceptions.

2016-07-12_05:14:55.58509 INFO  05:14:55 [main]: JOINING: Starting to bootstrap...
2016-07-12_05:14:56.07491 INFO  05:14:56 [GossipTasks:1]: InetAddress /2401:db00:2011:50c7:face:0:9:0 is now DOWN
2016-07-12_05:14:56.32219 Exception (java.lang.RuntimeException) encountered during startup: A node required to move the data consistently is down (/2401:db00:2011:50c7:face:0:9:0). If you wish to move the data from a potentially inconsis
tent replica, restart the node with -Dcassandra.consistent.rangemovement=false
2016-07-12_05:14:56.32582 ERROR 05:14:56 [main]: Exception encountered during startup
2016-07-12_05:14:56.32583 java.lang.RuntimeException: A node required to move the data consistently is down (/2401:db00:2011:50c7:face:0:9:0). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dc
assandra.consistent.rangemovement=false
2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32584       at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32584       at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32585       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:709) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32585       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32585       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32586       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32586       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_05:14:56.32730 WARN  05:14:56 [StorageServiceShutdownHook]: No local state or state is in silent shutdown, not announcing shutdown

Here are more logs: https://gist.github.com/DikangGu/c6a83eafdbc091250eade4a3bddcc40b

I'm pretty sure there are no DOWN nodes or restarted nodes in the cluster, but I still see a lot of nodes UP and DOWN in the gossip log, which failed the bootstrap at the end, is this a known bug?