[CASSANDRA-15812] Submitting Validation requests can block ANTI_ENTROPY stage

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta1, 4.0
• Component/s: Consistency/Repair
• Bug Category: Degradation - Resource Management
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/04533e6cdae94f91a62d769c874156d81301dc7d

Test and Documentation Plan

new unit tests added, existing dtests modified  

• Assignee: Sam Tunnicliffe
• Reporter: Sam Tunnicliffe

Authors

Sam Tunnicliffe

Reviewers

Benjamin Lerer

• Created: 13/May/20 19:03
• Updated: 21/Dec/20 07:58
• Resolved: 09/Jun/20 12:46

Description

 RepairMessages are handled on Stage.ANTI_ENTROPY, which has a thread pool with core/max capacity of one, ie. we can only process one message at a time. 
 
Scheduling validation compactions may however block the stage completely, by blocking on CompactionManager's ValidationExecutor while submitting a new validation compaction, in cases where there are already more validations running than can be executed in parallel.