[CASSANDRA-10844] failed_bootstrap_wiped_node_can_join_test is failing

• Type: Sub-task
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.13
• Component/s: Legacy/Streaming and Messaging, Legacy/Testing
• Assignee: Joel Knighton
• Reporter: Philip Thompson

Authors

Joel Knighton

Reviewers

Stefania Alborghetti

• Created: 11/Dec/15 17:13
• Updated: 16/Apr/19 09:30
• Resolved: 05/Jan/16 14:35

Description

bootstrap_test.TestBootstrap.failed_bootstap_wiped_node_can_join_test is failing on 2.1-head. The second node fails to join the cluster. I see a lot of exceptions in node1's log, such as

ERROR [STREAM-OUT-/127.0.0.2] 2015-12-11 12:06:13,778 StreamSession.java:505 - [Stream #7b5ec5a0-a029-11e5-bad9-ffd0922f40e6] Streaming error occurred
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_51]
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_51]
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_51]
        at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_51]
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_51]
        at org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48) ~[main/:na]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44) ~[main/:na]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351) [main/:na]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331) [main/:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

Which seem consistent with node2 being killed, so the bootstrap fails. But then when restarting node2, it does not join. It looks like it fails to rejoin because of a false positive in checking the 2 minute rule.

ERROR [main] 2015-12-11 12:06:17,954 CassandraDaemon.java:579 - Except
ion encountered during startup
java.lang.UnsupportedOperationException: Other bootstrapping/leaving/m
oving nodes detected, cannot bootstrap while cassandra.consistent.rang
emovement is true
        at org.apache.cassandra.service.StorageService.checkForEndpoin
tCollision(StorageService.java:559) ~[main/:na]
        at org.apache.cassandra.service.StorageService.prepareToJoin(S
torageService.java:789) ~[main/:na]
        at org.apache.cassandra.service.StorageService.initServer(Stor
ageService.java:721) ~[main/:na]
        at org.apache.cassandra.service.StorageService.initServer(Stor
ageService.java:612) ~[main/:na]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:387) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) [main/:na]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:651) [main/:na]

This fails consistently locally and on cassci. Logs attached.