[CASSANDRA-4411] Assertion with LCS compaction

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.1.3, 1.1.4
• Labels: lcs
• Severity: Normal
• Assignee: Sylvain Lebresne
• Reporter: Anton Winter

Authors

Sylvain Lebresne

Reviewers

Jonathan Ellis

• Created: 05/Jul/12 04:52
• Updated: 16/Apr/19 09:32
• Resolved: 16/Jul/12 15:33

Description

As instructed in CASSANDRA-4321 I have raised this issue as a continuation of that issue as it appears the problem still exists.

I have repeatedly run sstablescrub across all my nodes after the 1.1.2 upgrade until sstablescrub shows no errors. The exceptions described in CASSANDRA-4321 do not occur as frequently now but the integrity check still throws exceptions on a number of nodes. Once those exceptions occur compactionstats shows a large number of pending tasks with no progression afterwards.

ERROR [CompactionExecutor:150] 2012-07-05 04:26:15,570 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:150,1,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214)
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158)
        at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531)
        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254)
        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:978)
        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
        at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
        at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)