[CASSANDRA-11448] Running OOS should trigger the disk failure policy

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.14, 2.2.6, 3.0.5, 3.5
• Severity: Normal
• Assignee: Branimir Lambov
• Reporter: Brandon Williams

Authors

Branimir Lambov

Reviewers

Joshua McKenzie

• Created: 28/Mar/16 19:53
• Updated: 16/Apr/19 09:30
• Resolved: 01/Apr/16 15:59

Description

Currently when you run OOS, this happens:

ERROR [MemtableFlushWriter:8561] 2016-03-28 01:17:37,047  CassandraDaemon.java:229 - Exception in thread Thread[MemtableFlushWriter:8561,5,main]   java.lang.RuntimeException: Insufficient disk space to write 48 bytes 
    at org.apache.cassandra.io.util.DiskAwareRunnable.getWriteDirectory(DiskAwareRunnable.java:29) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
    at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:332) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.1.jar:na]
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120) ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]

Now your flush writer is dead and postflush tasks build up forever. Instead we should throw FSWE and trigger the failure policy.

Issue Links

• relates to: CASSANDRA-11828 Commit log needs to track unflushed intervals rather than positions