[CASSANDRA-9440] Bootstrap fails without any hint of prior stream failure

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem

Environment

Cassandra 2.0.14
2 DCs on EC2

• Severity: Normal
• Assignee: Unassigned
• Reporter: Omid Aladini
• Created: 20/May/15 15:45
• Updated: 16/Apr/19 09:31
• Resolved: 29/May/15 09:21

Description

I'm working on a cluster running Cassandra 2.0.14 and the bootstrap fails but there is no prior hint of failed streams:

 WARN [StreamReceiveTask:177] 2015-05-20 04:20:55,251 StreamResultFuture.java (line 215) [Stream #0b42c640-fe03-11e4-8a6f-dd5dc9b30af4] Stream failed
ERROR [main] 2015-05-20 04:20:55,252 CassandraDaemon.java (line 584) Exception encountered during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
        at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86)
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1005)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:808)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:621)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:510)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:437)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:656)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
        at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
        at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:216)
        at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:377)
        at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:662)
        at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:613)
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:143)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
 INFO [StorageServiceShutdownHook] 2015-05-20 04:20:55,286 Gossiper.java (line 1330) Announcing shutdown

There are no WARN or ERROR prior to this in the log files of the bootstrapping node or other nodes in the cluster. Only relevant log lines are "Session with 11.22.33.44/11.22.33.44 is complete"

Is it possible that individual stream sessions fail silently?