[CASSANDRA-19336] Repair causes out of memory

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.13, 4.1.5, 5.0-rc1, 5.0, 5.1
• Component/s: Consistency/Repair
• Bug Category: Availability - Cluster Crash
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.19

Source Control Link

https://github.com/apache/cassandra/commit/505f5af645c1712c3da42d98d005276396ff2667

Test and Documentation Plan

Patch	CI
4.0	j8 j11
4.1	j8 j11
5.0	j11 j17
trunk	j11 j17

• Assignee: Andres de la Peña
• Reporter: Andres de la Peña

Authors

Andres de la Peña

Reviewers

David Capwell

• Created: 29/Jan/24 13:45
• Updated: 26/Mar/24 10:38
• Resolved: 08/Feb/24 11:46

Description

CASSANDRA-14096 introduced repair_session_space as a limit for the memory usage for Merkle tree calculations during repairs. This limit is applied to the set of Merkle trees built for a received validation request (VALIDATION_REQ), divided by the replication factor so as not to overwhelm the repair coordinator, who will have requested RF sets of Merkle trees. That way the repair coordinator should only use repair_session_space for the RF Merkle trees.

However, a repair session without pr/-partitioner-range will send RF*RF validation requests, because the repair coordinator node has RF-1 replicas and is also the replica of RF-1 nodes. Since all the requests are sent at the same time, at some point the repair coordinator can have up to RF*repair_session_space worth of Merkle trees if none of the validation responses is fully processed before the last response arrives.

Even worse, if the cluster uses virtual nodes, many nodes can be replicas of the repair coordinator, and some nodes can be replicas of multiple token ranges. It would mean that the repair coordinator can send more than RF or RF*RF simultaneous validation requests.

For example, in an 11-node cluster with RF=3 and 256 tokens, we have seen a repair session involving 44 groups of ranges to be repaired. This produces 44*3=132 validation requests contacting all the nodes in the cluster. When the responses for all these requests start to arrive to the coordinator, each containing up to repair_session_space/3 of Merkle trees, they accumulate quicker than they are consumed, greatly exceeding repair_session_space and OOMing the node.

Issue Links

• is duplicated by: CASSANDRA-17787 Full repair on a keyspace with a large amount of tables causes OOM
• is related to: CASSANDRA-14096 Cassandra 3.11.1 Repair Causes Out of Memory
• is related to: CASSANDRA-17787 Full repair on a keyspace with a large amount of tables causes OOM
• Testing discovered: CASSANDRA-19376 CompactionStrategyManagerTest.testAutomaticUpgradeConcurrency is flaky
• links to: GitHub Pull Request #3073