[CASSANDRA-19329] Repair can trigger the disk_failure_policy

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.0.x, 4.1.x, 5.0.x, 5.x
• Component/s: Consistency/Repair
• Platform: All
• Assignee: Unassigned
• Reporter: Meredith Silvernail
• Created: 26/Jan/24 13:47
• Updated: 31/Jan/24 17:08

Description

While a cluster was running a repair, we observed Insufficient disk space would unexpectedly shut down gossip instead of skipping the repair and give the below error.  It would seem the right behavior would be to skip and go on to the next one, as nodetool garbagecollect and cleanup would do.  Instead the two nodes that were experiencing the error continued to stay in UP-PARTIAL state with no gossip running.

 

2024-01-20 18:37:50,716 [ERROR] [CompactionExecutor:2] cluster_id=9 ip_address=10.0.0.1 CassandraDaemon.java:581 - Exception in thread Thread[CompactionExecutor:2,1,main]
org.apache.cassandra.io.FSDiskFullWriteError: java.io.IOException: Insufficient disk space to write 625503268755 bytes into the foobar keyspace
    at org.apache.cassandra.db.Directories.getWriteableLocation(Directories.java:443)
    at org.apache.cassandra.db.Directories.getWriteableLocationAsFile(Directories.java:339)
    at org.apache.cassandra.db.compaction.CompactionManager.antiCompactGroup(CompactionManager.java:1556)
    at org.apache.cassandra.db.compaction.CompactionManager.doAntiCompaction(CompactionManager.java:1522)
    at org.apache.cassandra.db.compaction.CompactionManager.performAnticompaction(CompactionManager.java:788)
    at org.apache.cassandra.db.compaction.CompactionManager$9.runMayThrow(CompactionManager.java:702)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Insufficient disk space to write 625503268755 bytes into the foobar keyspace
    at org.apache.cassandra.io.FSDiskFullWriteError.<init>(FSDiskFullWriteError.java:27)
    ... 13 common frames omitted