Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13833

Failed compaction is not captured

    XMLWordPrintableJSON

    Details

    • Severity:
      Normal

      Description

      Follow up for CASSANDRA-13785, when the compaction failed, it fails silently. No error message is logged and exceptions metric is not updated. Basically, it's unable to get the exception: CompactionManager.java:1491

      Here is the call stack:

          at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
          at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
          at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
          at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
          at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
          at java.lang.Thread.run(Thread.java:745)
      

      There're 2 FutureTask in the call stack, for example FutureTask1(FutureTask2)), If the call thrown an exception, FutureTask2 sets the status, save the exception and return. But FutureTask1 doesn't get any exception, then set the status to normal. So we're unable to get the exception in:
      CompactionManager.java:1491

      2.1.x is working fine, here is the call stack:

          at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[main/:na]
          at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
          at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73) ~[main/:na]
          at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[main/:na]
          at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[main/:na]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_141]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
      

        Attachments

          Activity

            People

            • Assignee:
              jay.zhuang Jay Zhuang
              Reporter:
              jay.zhuang Jay Zhuang
              Authors:
              Jay Zhuang
              Reviewers:
              Marcus Eriksson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: