Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6845

Cleanup fails with IndexOutOfBoundsException exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.0.7
    • Component/s: None
    • Labels:
      None
    • Environment:

      CentOS Linux 2.6.32-431.el6.x86_64
      Datastax Cassandra 2.0.5-1

    • Severity:
      Normal

      Description

      Originally we had Cassandra cluster with the following configuration:

      Initial tokens:
      10.1.11.51: -9223372036854775808
      10.1.11.52: 0
      10.2.11.51: -9223372036854775807
      10.2.11.52: 1

      Endpoint snitch: PropertyFileSnitch
      10.1.11.51=DC1:RAC1
      10.1.11.52=DC1:RAC1
      10.2.11.51=DC2:RAC1
      10.2.11.52=DC2:RAC1

      We have created keyspace with network topology strategy and replication factor 2:

      CREATE KEYSPACE our_keyspace WITH replication = {
        'class': 'NetworkTopologyStrategy',
        'DC2': '2',
        'DC1': '2'
      };
      

      All column families have been created with SizeTiered compaction strategy. Аfter a while column families have been migrated to LeveledCompactionStrategy and then replication factor was decreased to 1.

      We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed with assertion error. Assertion stacktrace was the same as described in CASSANDRA-6774 issue.
      Then we've stopped all compactions, started cleanup again and got another exception:

      Error occurred during cleanup
      java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:188)
      at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
      at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
      at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
      at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
      at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
      at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
      at sun.rmi.transport.Transport$1.run(Transport.java:177)
      at sun.rmi.transport.Transport$1.run(Transport.java:174)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      at java.util.ArrayList.rangeCheck(ArrayList.java:635)
      at java.util.ArrayList.get(ArrayList.java:411)
      at org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
      at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
      at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
      at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
      at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      ... 3 more

      It seems there is typo in needsCleanup method of CompactionManager class and the supplied patch should fix the issue.
      Get method on the line 502 (CompactionManager.java) throws exception, because of the wrong comparison on the line 496: there should be sortedRanges size check, not ownedRanges.

        Attachments

          Activity

            People

            • Assignee:
              dimchansky Dmitrij Koniajev
              Reporter:
              dimchansky Dmitrij Koniajev
              Authors:
              Dmitrij Koniajev
              Reviewers:
              Marcus Eriksson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: