[CASSANDRA-13704] Safer handling of out of range tokens

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.0.14, 4.1.7, 5.0.1
• Component/s: Legacy/Coordination, Legacy/Observability
• Severity: Critical
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/b4f1c6d250991fc27c5f87a2dc0fd81e7ad0d272

Test and Documentation Plan

There are several new unit and distributed tests around out-of-token-range requests, in addition to new bits of logging when these requests occur and metrics that count their frequencies.

• Assignee: Caleb Rackliffe
• Reporter: Sam Tunnicliffe

Authors

Ariel Weisberg, Caleb Rackliffe, Michael Semb Wever, Sam Tunnicliffe

Reviewers

Michael Semb Wever

• Created: 20/Jul/17 15:34
• Updated: 17/Sep/24 22:30
• Resolved: 17/Sep/24 22:29

Description

It is possible for nodes to have a divergent view of the ring, which can result in some operations being sent to the wrong nodes. This is an umbrella ticket to mitigate such issues by adding logging when a node is asked to perform an operation for tokens it does not own. This will be useful for detecting when the nodes' views of the ring diverge, which is not highly visible at the moment, and also for post-hoc analysis.

It may also be beneficial to straight up reject certain operations, though this will need to balance the risk of performing those ops against the consequences rejecting them has on availability.

Issue Links

• incorporates: CASSANDRA-13705 Warn or error when receiving write requests for unowned token ranges
• incorporates: CASSANDRA-13706 Warn or error when receiving a read request for unowned token ranges
• incorporates: CASSANDRA-13707 Warn or error when receiving Merkle Tree requests for unowned token ranges
• incorporates: CASSANDRA-13708 Warn or error when receiving stream requests or inbound streams that contain unowned token ranges
• incorporates: CASSANDRA-13709 Warn or error when receiving hints for unowned token ranges
• incorporates: CASSANDRA-13710 Add a nodetool command to display details of any tables containing unowned tokens
• links to: GitHub Pull Request #3526
• links to: GitHub Pull Request #3539
• links to: GitHub Pull Request #3544

Sub-Tasks

• Warn or error when receiving write requests for unowned token ranges: Resolved
• Warn or error when receiving a read request for unowned token ranges: Resolved
• Warn or error when receiving Merkle Tree requests for unowned token ranges: Resolved
• Warn or error when receiving stream requests or inbound streams that contain unowned token ranges: Resolved
• Warn or error when receiving hints for unowned token ranges: Resolved
• Add a nodetool command to display details of any tables containing unowned tokens: Resolved