Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6845

Cleanup fails with IndexOutOfBoundsException exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.7
    • None
    • None
    • CentOS Linux 2.6.32-431.el6.x86_64
      Datastax Cassandra 2.0.5-1

    • Normal

    Description

      Originally we had Cassandra cluster with the following configuration:

      Initial tokens:
      10.1.11.51: -9223372036854775808
      10.1.11.52: 0
      10.2.11.51: -9223372036854775807
      10.2.11.52: 1

      Endpoint snitch: PropertyFileSnitch
      10.1.11.51=DC1:RAC1
      10.1.11.52=DC1:RAC1
      10.2.11.51=DC2:RAC1
      10.2.11.52=DC2:RAC1

      We have created keyspace with network topology strategy and replication factor 2:

      CREATE KEYSPACE our_keyspace WITH replication = {
        'class': 'NetworkTopologyStrategy',
        'DC2': '2',
        'DC1': '2'
      };
      

      All column families have been created with SizeTiered compaction strategy. Аfter a while column families have been migrated to LeveledCompactionStrategy and then replication factor was decreased to 1.

      We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed with assertion error. Assertion stacktrace was the same as described in CASSANDRA-6774 issue.
      Then we've stopped all compactions, started cleanup again and got another exception:

      Error occurred during cleanup
      java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
      at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
      at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
      at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
      at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
      at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
      at sun.rmi.transport.Transport$1.run(Transport.java:177)
      at sun.rmi.transport.Transport$1.run(Transport.java:174)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      at java.util.ArrayList.rangeCheck(ArrayList.java:635)
      at java.util.ArrayList.get(ArrayList.java:411)
      at org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
      at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
      at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
      at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
      at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      ... 3 more

      It seems there is typo in needsCleanup method of CompactionManager class and the supplied patch should fix the issue.
      Get method on the line 502 (CompactionManager.java) throws exception, because of the wrong comparison on the line 496: there should be sortedRanges size check, not ownedRanges.

      Attachments

        Activity

          People

            dimchansky Dmitrij Koniajev
            dimchansky Dmitrij Koniajev
            Dmitrij Koniajev
            Marcus Eriksson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: