[CASSANDRA-10270] Cassandra stops compacting

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.2.2, 3.0.0 rc1
• Component/s: Local/Compaction

Environment

linux (google cloud click-to-deploy, default settings, 3 nodes)

• Severity: Critical
• Assignee: Marcus Eriksson
• Reporter: Adam Bliss

Authors

Marcus Eriksson

Reviewers

Benedict Elliott Smith

• Created: 04/Sep/15 21:11
• Updated: 16/Apr/19 09:30
• Resolved: 10/Sep/15 06:40

Description

My cassandra cluster won't keep compacting. I notice that if I restart, it does compact for a while, but after a time it stops. As a result, after adding a bunch of rows, I ended up with about 1000 sstables per node.

I'll attach more logs in a minute, but it seems like this might be the most relevant part:

INFO  [CompactionExecutor:1] 2015-09-04 14:22:55,796 CompactionManager.java:1433 - Compaction interrupted: Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2, ranks_by_domain, 812501702/7543091905)bytes
DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 CompactionManager.java:1437 - Full interruption stack trace:
org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted: Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2, ranks_by_
domain, 812501702/7543091905)bytes
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:180) ~[apache-cassandra-2.2.1.jar:2.2.1]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.1.jar:2.2.1]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.1.jar:2.2.1]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.1.jar:2.2.1]                        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236) ~[apache-cassandra-2.2.1.jar:2.2.1]                                                                                                                                                                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]                                                                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79]                                                               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]                                                                at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 CompactionManager.java:222 - Checking system.local                                                         DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 SizeTieredCompactionStrategy.java:85 - Compaction buckets are [[BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-83-big-Data.db'), BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-81-big-Data.db'),