[CASSANDRA-8716] "java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed" when running cleanup

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.0.13
• Labels: qa-resolved

Environment

Centos 6.6, Cassandra 2.0.12, Oracle JDK 1.7.0_67

• Severity: Low
• Assignee: Robert Stupp Assign to me
• Reporter: Imri Zvik

Authors

Robert Stupp

Reviewers

Benedict Elliott Smith

• Tester: Philip Thompson
• Created: 02/Feb/15 13:32
• Updated: 16/Apr/19 09:31
• Resolved: 17/Feb/15 16:29

Description

Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
        at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
        at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Memory was freed
        at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
        at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
        at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
        at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
        at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
        at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
        at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
        at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
        at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
        at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
        at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
        at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more

The error continue to happen even after scrub.
If I retry a few time, it may run successfully.
I am attaching the server log.

Issue Links

• is broken by: CASSANDRA-7386 JBOD threshold to prevent unbalanced disk utilization