[CASSANDRA-16800] Abort repairs whose streaming jobs will push disk usage over a configurable threshold

• Type: Improvement
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: Consistency/Repair, Consistency/Streaming
• Change Category: Operability
• Complexity: Normal
• Platform: All
• Assignee: Unassigned
• Reporter: Caleb Rackliffe
• Created: 13/Jul/21 19:17
• Updated: 07/Mar/23 10:53

Description

When nodes are out of sync, it is possible to get into a situation where streaming from repairs sends us to 100% disk usage (from a starting point of 20% even, in matter of hours). Since we know the size of the data we're going to stream over, and how much disk space is left, we should just fail the streaming instead of causing flush issues on memtables/commitlog/etc. as we approach that point.

Perhaps it would make sense to have a configurable threshold of say 90% disk usage over which we won't accept more streams.