Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
None
-
None
-
Cassandra 2.1.18
-
Normal
Description
Hi,
We have a Cassandra DC with 31 nodes,but it's always failed when adding the 32nd node.
The following is the error log.And we suspect it's due to the secondary index,but it never happens before.Could you please advise?
Thanks
INFO [CompactionExecutor:3] 2017-07-24 22:07:40,774 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 149 (0%) on-heap, 20 (0%) off-heap INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:347 - Writing Memtable-compactions_in_progress@581635159(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:382 - Completed flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1159-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=11929032) INFO [CompactionExecutor:3] 2017-07-24 22:07:40,796 CompactionTask.java:274 - Compacted 15 sstables to [/data/cassandra/data/ecommercedata/ecommerce_baitiao_bill-c1a7f8d0660511e781e2077cc592d79d/ecommercedata-ecommerce_baitiao_bill-ka-40,]. 293,793 bytes to 288,251 (~98% of original) in 108ms = 2.545348MB/s. 1,024 total partitions merged to 1,016. Partition merge counts were {1:1008, 2:8, } INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,351 ColumnFamilyStore.java:917 - Enqueuing flush of ecommerce_baitiao_record.baitiao_state: 2228978 (0%) on-heap, 6183142 (0%) off-heap INFO [CompactionExecutor:3] 2017-07-24 22:07:43,352 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 349 (0%) on-heap, 266 (0%) off-heap INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-ecommerce_baitiao_record.baitiao_state@1651345625(4.792MiB serialized bytes, 89948 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-compactions_in_progress@284311350(0.197KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:382 - Completed flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1160-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501) INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,542 Memtable.java:382 - Completed flushing /data/cassandra/data/ecommercedata/ecommerce_baitiao_record-c94692e012cd11e682501f0d5e9f1e54/ecommercedata-ecommerce_baitiao_record.baitiao_state-tmp-ka-15-Data.db (1.593MiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501) INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558 SecondaryIndexManager.java:174 - Index build of [ecommerce_baitiao_record.baitiao_state] complete INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558 StreamResultFuture.java:181 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293] Session with /xxx.xxx.xxx.xxx is complete WARN [StreamReceiveTask:5] 2017-07-24 22:07:43,559 StreamResultFuture.java:208 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293] Stream failed ERROR [main] 2017-07-24 22:07:43,561 CassandraDaemon.java:583 - Exception encountered during startup java.lang.RuntimeException: Error during boostrap: Stream failed at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18] Caused by: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.18.jar:2.1.18] at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661) ~[apache-cassandra-2.1.18.jar:2.1.18] at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.18.jar:2.1.18] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] WARN [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 Gossiper.java:1462 - No local state or state is in silent shutdown, not announcing shutdown INFO [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 MessagingService.java:734 - Waiting for messaging service to quiesce INFO [ACCEPT-/172.16.135.52] 2017-07-24 22:07:43,565 MessagingService.java:1020 - MessagingService has terminated the accept() thread INFO [CompactionExecutor:3] 2017-07-24 22:07:43,571 CompactionTask.java:141 - Compacting [SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-10-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-12-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-11-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-14-Data.db')]