[CASSANDRA-11201] Compaction memory fault in 3.0.3

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Local/Compaction

Environment

debian jesse latest release, updated Feb. 20th

• Severity: Normal
• Assignee: Benjamin Lerer
• Reporter: Jason Kania

Authors

Benjamin Lerer

• Created: 20/Feb/16 23:50
• Updated: 21/Dec/21 02:19
• Resolved: 21/Jun/17 08:38

Description

I have been encountering the following errors periodically on the system:

ERROR [CompactionExecutor:6] 2016-02-20 16:54:09,069 CassandraDaemon.java:195 - Exception in thread Thread[CompactionExecutor:6,1,main]
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:376) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:175) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:280) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.3.jar:3.0.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]

This problem persisted after several reboots and even when most other applications on the system were terminated to provide more memory availability.

The problem also occurs when running 'nodetool compact'.

Issue Links

• is duplicated by: CASSANDRA-11200 CompactionExecutor thread error brings down JVM in 3.0.3
• relates to: CASSANDRA-17223 Node can be left in a bad state after invalid memory access exception