Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13833

Failed compaction is not captured

    XMLWordPrintableJSON

Details

    • Normal

    Description

      Follow up for CASSANDRA-13785, when the compaction failed, it fails silently. No error message is logged and exceptions metric is not updated. Basically, it's unable to get the exception: CompactionManager.java:1491

      Here is the call stack:

          at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
          at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
          at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
          at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
          at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
          at java.lang.Thread.run(Thread.java:745)
      

      There're 2 FutureTask in the call stack, for example FutureTask1(FutureTask2)), If the call thrown an exception, FutureTask2 sets the status, save the exception and return. But FutureTask1 doesn't get any exception, then set the status to normal. So we're unable to get the exception in:
      CompactionManager.java:1491

      2.1.x is working fine, here is the call stack:

          at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[main/:na]
          at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
          at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73) ~[main/:na]
          at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[main/:na]
          at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[main/:na]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_141]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_141]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
      

      Attachments

        Activity

          People

            jay.zhuang Jay Zhuang
            jay.zhuang Jay Zhuang
            Jay Zhuang
            Marcus Eriksson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: