[CASSANDRA-15433] Pending ranges are not recalculated on keyspace creation

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.26, 3.11.12, 4.0.2, 4.1-alpha1, 4.1
• Component/s: Cluster/Membership
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/11af037fd99bfb4a942f7d7dd55c177a37d29f63

Test and Documentation Plan

in-jvm dtest

 

• Assignee: Sumanth Pasupuleti
• Reporter: Josh Snyder

Authors

Sumanth Pasupuleti

Reviewers

Alex Petrov

• Created: 19/Nov/19 01:55
• Updated: 27/May/22 19:25
• Resolved: 15/Oct/21 09:30

Description

When a node begins bootstrapping, Cassandra recalculates pending tokens for each keyspace that exists when the state change is observed (in StorageService:handleState*). When new keyspaces are created, we do not recalculate pending ranges (around Schema:merge). As a result, writes for new keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When bootstrapping finishes, the node which just bootstrapped will not have data for the newly created keyspace.

Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and when it finishes bootstrapping, C will cede ownership of some ranges to D. A quorum write is acknowledged by C and A. B missed the write, and the coordinator didn't send it to D at all. When D finishes bootstrapping, the quorum B+D will not contain the mutation.

Steps to reproduce:

1. Join a node in BOOT mode
2. Create a keyspace
3. Send writes to that keyspace
4. On the joining node, observe that nodetool cfstats records zero writes to the new keyspace

I have observed this directly in Cassandra 3.0, and based on my reading the code, I believe it affects up through trunk.

Issue Links

• depends upon: CASSANDRA-15497 Implement node bootstrap in in-JVM tests