[CASSANDRA-13229] dtest failure in topology_test.TestTopology.size_estimates_multidc_test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.7, 4.0-alpha1, 4.0
• Component/s: Test/dtest/python
• Labels: dtest test-failure
• Severity: Normal
• Assignee: Alex Petrov
• Reporter: Sean McCarthy

Authors

Alex Petrov

Reviewers

Paulo Motta

• Created: 16/Feb/17 15:43
• Updated: 01/Aug/21 11:17
• Resolved: 12/May/17 06:54

Description

example failure:

http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test

Standard Output

Unexpected error in node1 log, error: 
ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211 - Exception in thread Thread[MemtablePostFlush:1,5,main]
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92) ~[main/:na]
	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180) ~[main/:na]
	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115) ~[main/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81) [main/:na]
	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown Source) [main/:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Unexpected error in node1 log, error: 
ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 - Exception in thread Thread[MigrationStage:1,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown Source) ~[na:na]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265) ~[main/:na]
	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402) ~[main/:na]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81) [main/:na]
	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown Source) [main/:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
	... 16 common frames omitted
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92) ~[main/:na]
	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180) ~[main/:na]
	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115) ~[main/:na]
	... 5 common frames omitted
Unexpected error in node1 log, error: 
ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
	at org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384) ~[main/:na]
	at org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176) ~[main/:na]
	at org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066) ~[main/:na]
	at org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091) ~[main/:na]
	at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048) ~[main/:na]
	at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043) ~[main/:na]
	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966) ~[main/:na]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:649) ~[main/:na]
	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581) ~[main/:na]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364) [main/:na]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [main/:na]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646) [main/:na]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
	... 12 common frames omitted
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown Source) ~[na:na]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
	at org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284) ~[main/:na]
	at org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265) ~[main/:na]
	at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
	at org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564) ~[main/:na]
	at org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402) ~[main/:na]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81) ~[main/:na]
	at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown Source) ~[na:na]
	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) ~[main/:na]
	... 16 common frames omitted
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
	at org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92) ~[main/:na]
	at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59) ~[main/:na]
	at org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180) ~[main/:na]
	at org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312) ~[main/:na]
	at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150) ~[main/:na]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115) ~[main/:na]
	... 5 common frames omitted
Unexpected error in node1 log, error: 
ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972 AbstractCommitLogSegmentManager.java:311 - Failed to force-recycle all segments; at least one segment is still in use with dirty CFs.

Issue Links

• is broken by: CASSANDRA-9639 size_estimates is inacurate in multi-dc clusters