[CASSANDRA-15084] failed compactions in specific cluster version 3.11.2

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 3.11.5
• Bug Category: Availability - Process Crash
• Severity: Normal
• Assignee: Unassigned
• Reporter: Tomer Yuval
• Created: 15/Apr/19 13:50
• Updated: 01/Aug/21 11:28
• Resolved: 23/Apr/19 08:22

Description

We are using Cassandra version 3.11.2

In our biggest cluster, in the biggest table compactions keep failing.

The amount of data in every node is around 900GB.

After we run the compaction, we monitor the process using nodetool compactionstats and can see the progress continue until it reached 99.99% of the progress.

In this point the compaction fail with the error below

In addition, we increased index_summary_capacity_in_mb few times until the size of 4GB, but with no help.

 
ERROR [CompactionExecutor:26976] 2019-04-11 13:55:24,917 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:26976,1,main]
java.lang.IllegalArgumentException: null
        at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_144]
        at org.apache.cassandra.io.util.SafeMemoryWriter.reallocate(SafeMemoryWriter.java:59) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.SafeMemoryWriter.setCapacity(SafeMemoryWriter.java:68) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.IndexSummaryBuilder.prepareToCommit(IndexSummaryBuilder.java:250) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:524) ~[apache-cassandra-3.11.2.jar:3.11.
2]
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.2.
jar:3.11.2]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364) ~[apache-cassandra-3.11.2.ja
r:3.11.2]
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.2.
jar:3.11.2]
        at org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:379) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.2.
jar:3.11.2]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:112) ~[apache-cassandra-3.11.2.jar:3.1
1.2]
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.2.
jar:3.11.2]
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) ~[apache-cassandra-3.11.2.jar:3.11.
2]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:122) ~[apache-cassandra-3.11.2.jar:3.11.2
]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:746) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_144]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]

Issue Links

• duplicates: CASSANDRA-14649 Index summaries fail when their size gets > 2G and use more space than necessary