I think it will be much better if we can prioritize, long running compaction vs normal compaction, lets sat we have
10MB Compaction limit
2MB Validation compaction limit
2MB is the limit for the validation for a while and when normal compaction kicks in we might want to hold the validation and do the compction complete because that will affect the read performance and continue with the validation compaction after that. by doing this we can set something like
12MB Compaction limit
6 MB Validation compaction limit
and still be within the HDD limit of 12MB.
The good thing about normal compaction is that it is spread out and not all the nodes are not involved in it.
I am starting to think that we can do repairs one by one for a range (within a region), so the traffic doesnt get stuck waiting for the IO. Hope it makes sense.