[CASSANDRA-13833] Failed compaction is not captured

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.2.11, 3.0.15, 3.11.1, 4.0-alpha1, 4.0
• Component/s: Local/Compaction
• Severity: Normal
• Assignee: Jay Zhuang
• Reporter: Jay Zhuang

Authors

Jay Zhuang

Reviewers

Marcus Eriksson

• Created: 31/Aug/17 17:39
• Updated: 07/Mar/23 11:52
• Resolved: 04/Sep/17 13:05

Description

Follow up for CASSANDRA-13785, when the compaction failed, it fails silently. No error message is logged and exceptions metric is not updated. Basically, it's unable to get the exception: CompactionManager.java:1491

Here is the call stack:

    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
    at java.lang.Thread.run(Thread.java:745)

There're 2 FutureTask in the call stack, for example FutureTask1(FutureTask2)), If the call thrown an exception, FutureTask2 sets the status, save the exception and return. But FutureTask1 doesn't get any exception, then set the status to normal. So we're unable to get the exception in:
CompactionManager.java:1491

2.1.x is working fine, here is the call stack:

    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[main/:na]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73) ~[main/:na]
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[main/:na]
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[main/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_141]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]