[CASSANDRA-7386] JBOD threshold to prevent unbalanced disk utilization

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.0.12, 2.1.3
• Assignee: Robert Stupp Assign to me
• Reporter: Chris Lohfink

Authors

Robert Stupp

Reviewers

Yuki Morishita

• Tester: Alan Boudreault
• Created: 12/Jun/14 18:00
• Updated: 16/Apr/19 09:31
• Resolved: 20/Nov/14 16:34

Description

Currently the pick the disks are picked first by number of current tasks, then by free space. This helps with performance but can lead to large differences in utilization in some (unlikely but possible) scenarios. Ive seen 55% to 10% and heard reports of 90% to 10% on IRC. With both LCS and STCS (although my suspicion is that STCS makes it worse since harder to be balanced).

I purpose the algorithm change a little to have some maximum range of utilization where it will pick by free space over load (acknowledging it can be slower). So if a disk A is 30% full and disk B is 5% full it will never pick A over B until it balances out.

Issue Links

• breaks: CASSANDRA-8716 "java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed" when running cleanup
• is duplicated by: CASSANDRA-7615 Data isn't written to the disk with enough space while using multiple data_file_directories
• is duplicated by: CASSANDRA-8571 Free space management does not work very well
• is related to: CASSANDRA-8329 LeveledCompactionStrategy should split large files across data directories when compacting
• links to: Git branch (2.0)
• links to: Git branch (2.1)