Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11448

Running OOS should trigger the disk failure policy

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.1.14, 2.2.6, 3.0.5, 3.5
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Currently when you run OOS, this happens:

      ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047  CassandraDaemon.java:229 - Exception in thread Thread[MemtableFlushWriter:8561,5,main]   java.lang.RuntimeException: Insufficient disk space to write 48 bytes 
          at org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
          at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
          at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
          at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:na]
          at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_66]
          at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
      

      Now your flush writer is dead and postflush tasks build up forever. Instead we should throw FSWE and trigger the failure policy.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              blambov Branimir Lambov Assign to me
              Reporter:
              brandon.williams Brandon Williams
              Authors:
              Branimir Lambov
              Reviewers:
              Joshua McKenzie

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment