Cassandra
  1. Cassandra
  2. CASSANDRA-2307

IndexOutOfBoundsException during compaction

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned to Cassandra, JRE 1.6.0_24

      Description

      We're getting an IndexOutOfBounds exception when compacting.

      Here's the detailed error we get on-screen when running "nodetool -h 10.3.133.10 compact":

      Error occured while compacting keyspace test
      java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
      at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
      at java.util.concurrent.FutureTask.get(Unknown Source)
      at org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
      at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
      at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
      at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
      at sun.rmi.transport.Transport$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Unknown Source)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.IndexOutOfBoundsException
      at java.nio.Buffer.checkIndex(Unknown Source)
      at java.nio.HeapByteBuffer.getInt(Unknown Source)
      at org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
      at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:94)
      at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
      at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
      at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
      at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
      at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
      at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
      at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      ... 3 more

      And here's the error I'm getting in my log file:

      ERROR [CompactionExecutor:1] 2011-03-09 19:16:52,299 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
      java.lang.IndexOutOfBoundsException
      at java.nio.Buffer.checkIndex(Unknown Source)
      at java.nio.HeapByteBuffer.getInt(Unknown Source)
      at org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
      at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
      at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:94)
      at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
      at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
      at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
      at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
      at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
      at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
      at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
      at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      We run Cassandra 0.7.3, but we had the issue with Cassandra 0.7.2 as well. We have 8 machines in the cluster, the error happens only on one machine, at least for now.

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Mike Nadeau
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development