[CASSANDRA-15669] LeveledCompactionStrategy compact last level throw an ArrayIndexOutOfBoundsException

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.11, 4.0-rc2, 4.0
• Component/s: Local/Compaction/LCS
• Bug Category: Availability
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.11.0

Source Control Link

https://github.com/apache/cassandra/commit/803a23b2f1b24701c22e7fadaf98c2c9a9491d35

Test and Documentation Plan

test added

• Assignee: Aleksei Zotov
• Reporter: sunhaihong

Authors

Aleksei Zotov

Reviewers

Marcus Eriksson, Yifan Cai

• Created: 27/Mar/20 13:58
• Updated: 12/Jun/21 10:17
• Resolved: 31/May/21 18:24

Description

Cassandra will throw an ArrayIndexOutOfBoundsException when compact last level.

My test is as follows：

1. Create a table with LeveledCompactionStrategy and its params are 'enabled': 'true', 'fanout_size': '2', 'max_threshold': '32', 'min_threshold': '4', 'sstable_size_in_mb': '2'（fanout_size and sstable_size_in_mb are too small just to make it easier to reproduce the problem）;
2. Insert data into the table by stress;
3. Cassandra throw an ArrayIndexOutOfBoundsException when compact level9 sstables(this level score bigger than 1.001)

ERROR [CompactionExecutor:4] 2020-03-28 08:59:00,990 CassandraDaemon.java:442 - Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.cassandra.db.compaction.LeveledManifest.getLevel(LeveledManifest.java:814)
at org.apache.cassandra.db.compaction.LeveledManifest.getCandidatesFor(LeveledManifest.java:746)
at org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:398)
at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:131)
at org.apache.cassandra.db.compaction.CompactionStrategyHolder.lambda$getBackgroundTaskSuppliers$0(CompactionStrategyHolder.java:109)
at org.apache.cassandra.db.compaction.AbstractStrategyHolder$TaskSupplier.getTask(AbstractStrategyHolder.java:66)
at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:214)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:289)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

I tested it on cassandra version 3.11.3 & 4.0-alpha3. The exception all happened.

once it triggers, level1- leveln compaction no longer works, level0 is still valid