[CASSANDRA-16245] Implement repair quality test scenarios

• Type: Task
• Status: Patch Available
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.0.x
• Component/s: Test/dtest/java
• Epic Link: 40_Quality_Test
• Change Category: Quality Assurance
• Complexity: Challenging
• Platform: All

Test and Documentation Plan

Perform repairs for a 3 nodes cluster using m5ad.xlarge instances.
Repaired keyspaces will use RF=3 or RF=2 (the latter is for subranges with different sets of replicas).

Mode	Version	Settings	Checks
Full repair	trunk	Sequential + All token ranges	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Full repair	trunk	Parallel + Primary range	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Full repair	trunk	Force terminate repair shortly after it was triggered	Repair threads must be cleaned up
Subrange repair	trunk	Sequential + single token range	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Subrange repair	trunk	Parallel + 10 token ranges which have the same replicas	"No anticompaction (repairedAt == 0) Out of sync ranges > 0 Subsequent run must show no out of sync range A single repair session will handle all subranges at once"
Subrange repair	trunk	Parallel + 10 token ranges which have different replicas	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range More than one repair session is triggered to process all subranges"
Incremental repair	trunk	"Parallel (mandatory) No compaction during repair"	"Anticompaction status (repairedAt != 0) on all SSTables No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously) Out of sync ranges > 0 + Subsequent run must show no out of sync range"
Incremental repair	trunk	"Parallel (mandatory) Major compaction triggered during repair"	"Anticompaction status (repairedAt != 0) on all SSTables No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously) Out of sync ranges > 0 + Subsequent run must show no out of sync range"
Incremental repair	trunk	Force terminate repair shortly after it was triggered.	Repair threads must be cleaned up

• Assignee: Radovan Zvoncek
• Reporter: Alexander Dejanovski

Authors

Alexander Dejanovski, Radovan Zvoncek

• Created: 05/Nov/20 08:53
• Updated: 12/Jun/21 07:52

Description

Implement the following test scenarios in a new test suite for repair integration testing with significant load:

Generate/restore a workload of ~100GB per node. Medusa should be considered to create the initial backup which could then be restored from an S3 bucket to speed up node population.
Data should on purpose require repair and be generated accordingly.

Perform repairs for a 3 nodes cluster with 4 cores each and 16GB-32GB RAM (m5d.xlarge instances would be the most cost efficient type).
Repaired keyspaces will use RF=3 or RF=2 in some cases (the latter is for subranges with different sets of replicas).

Mode	Version	Settings	Checks
Full repair	trunk	Sequential + All token ranges	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Full repair	trunk	Parallel + Primary range	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Full repair	trunk	Force terminate repair shortly after it was triggered	Repair threads must be cleaned up
Subrange repair	trunk	Sequential + single token range	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range"
Subrange repair	trunk	Parallel + 10 token ranges which have the same replicas	"No anticompaction (repairedAt == 0) Out of sync ranges > 0 Subsequent run must show no out of sync range A single repair session will handle all subranges at once"
Subrange repair	trunk	Parallel + 10 token ranges which have different replicas	"No anticompaction (repairedAt==0) Out of sync ranges > 0 Subsequent run must show no out of sync range More than one repair session is triggered to process all subranges"
Incremental repair	trunk	"Parallel (mandatory) No compaction during repair"	"Anticompaction status (repairedAt != 0) on all SSTables No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously) Out of sync ranges > 0 + Subsequent run must show no out of sync range"
Incremental repair	trunk	"Parallel (mandatory) Major compaction triggered during repair"	"Anticompaction status (repairedAt != 0) on all SSTables No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously) Out of sync ranges > 0 + Subsequent run must show no out of sync range"
Incremental repair	trunk	Force terminate repair shortly after it was triggered.	Repair threads must be cleaned up

Issue Links

• is a child of: CASSANDRA-15580 4.0 quality testing: Repair