[CASSANDRA-6845] Cleanup fails with IndexOutOfBoundsException exception

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.7

Environment

CentOS Linux 2.6.32-431.el6.x86_64
Datastax Cassandra 2.0.5-1

• Severity: Normal
• Assignee: Dmitrij Koniajev
• Reporter: Dmitrij Koniajev

Authors

Dmitrij Koniajev

Reviewers

Marcus Eriksson

• Created: 12/Mar/14 16:05
• Updated: 16/Apr/19 09:31
• Resolved: 12/Mar/14 17:19

Description

Originally we had Cassandra cluster with the following configuration:

Initial tokens:
10.1.11.51: -9223372036854775808
10.1.11.52: 0
10.2.11.51: -9223372036854775807
10.2.11.52: 1

Endpoint snitch: PropertyFileSnitch
10.1.11.51=DC1:RAC1
10.1.11.52=DC1:RAC1
10.2.11.51=DC2:RAC1
10.2.11.52=DC2:RAC1

We have created keyspace with network topology strategy and replication factor 2:

CREATE KEYSPACE our_keyspace WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'DC2': '2',
  'DC1': '2'
};

All column families have been created with SizeTiered compaction strategy. Аfter a while column families have been migrated to LeveledCompactionStrategy and then replication factor was decreased to 1.

We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed with assertion error. Assertion stacktrace was the same as described in CASSANDRA-6774 issue.
Then we've stopped all compactions, started cleanup again and got another exception:

Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more

It seems there is typo in needsCleanup method of CompactionManager class and the supplied patch should fix the issue.
Get method on the line 502 (CompactionManager.java) throws exception, because of the wrong comparison on the line 496: there should be sortedRanges size check, not ownedRanges.