[CASSANDRA-15928] Throw FSWriteError upon write failures in order to apply DiskFailurePolicy

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta2, 4.0
• Component/s: Local/Other
• Bug Category: Degradation - Resource Management
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: Code Inspection
• Platform: All
• Since Version: 3.0.0

Source Control Link

[0e65e85861b84e87691d576ba80c5c3b3b7020d5|https://github.com/apache/cassandra/commit/0e65e85861b84e87691d576ba80c5c3b3b7020d5

Test and Documentation Plan

ci

• Assignee: Yifan Cai
• Reporter: Yifan Cai

Authors

Yifan Cai

Reviewers

Aleksey Yeschenko

• Created: 07/Jul/20 22:13
• Updated: 03/Jan/21 17:03
• Resolved: 03/Aug/20 15:37

Description

Active handling of FSError in AbstractLocalAwareExecutorService was introduced in CASSANDRA-14993. 
 
However, in the current code base, there are places that catch the IOException but not wrap it with FSError, hence not triggering the DiskFailurePolicy. Especially when the bad disk no longer permits write operations, it could leads to the mutation stage backup. Therefore I propose to fix the IOException handling in those cases. 
 
From the code inspection, those 6 places are current not re-throwing an IOException with FSWriteError.

1. org.apache.cassandra.triggers.CustomClassLoader#addClassPath throws IOError. Invoked in TriggerExecutor's constructor
2. org.apache.cassandra.io.util.FileUtils#renameWithConfirm throws RuntimeException
3. org.apache.cassandra.io.util.FileUtils#truncate throws RuntimeException
4. org.apache.cassandra.io.util.FileUtils#write throws RuntimeException
5. org.apache.cassandra.db.compaction.LeveledManifest#sendBackToL0 throws RuntimeException. Invokes rewriteSSTableMetadata
6. org.apache.cassandra.io.sstable.SSTableHeaderFix#writeNewMetadata throws RuntimeException. Invokes rewriteSSTableMetadata