[CASSANDRA-13545] Exception in CompactionExecutor leading to tmplink files not being removed

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Component/s: Local/Compaction
• Severity: Critical
• Since Version: 2.2.5
• Assignee: Unassigned
• Reporter: Dmitry Erokhin
• Created: 22/May/17 15:54
• Updated: 16/Apr/19 09:30
• Resolved: 24/Jan/19 18:44

Description

We are facing an issue where compactions fail on a few nodes with the following message

ERROR [CompactionExecutor:1248] 2017-05-22 15:32:55,390 CassandraDaemon.java:185 - Exception in thread Thread[CompactionExecutor:1248,1,main]
java.lang.AssertionError: null
	at org.apache.cassandra.io.sstable.IndexSummary.<init>(IndexSummary.java:86) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.io.sstable.IndexSummaryBuilder.build(IndexSummaryBuilder.java:235) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:316) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:170) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:115) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.append(DefaultCompactionWriter.java:64) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:184) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:256) ~[apache-cassandra-2.2.5.jar:2.2.5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

Also, the number of tmplink files in /var/lib/cassandra/data/<keyspace name>/blocks/tmplink* is growing constantly until node runs out of space. Restarting cassandra removes all tmplink files, but the issue still continues.
We are using Cassandra 2.2.5 on Debian 8 with Oracle Java 8

root@cassandra-p10:/var/lib/cassandra/data/mugenstorage/blocks-33167ef0447a11e68f3e5b42fc45b62f# dpkg -l | grep -E "java|cassandra"
ii  cassandra                      2.2.5                        all          distributed storage system for structured data
ii  cassandra-tools                2.2.5                        all          distributed storage system for structured data
ii  java-common                    0.52                         all          Base of all Java packages
ii  javascript-common              11                           all          Base support for JavaScript library packages
ii  oracle-java8-installer         8u121-1~webupd8~0            all          Oracle Java(TM) Development Kit (JDK) 8
ii  oracle-java8-set-default       8u121-1~webupd8~0            all          Set Oracle JDK 8 as default Java

Issue Links

• relates to: CASSANDRA-12743 Assertion error while running compaction