[CASSANDRA-16692] Unable to replace node with stale schema

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.25, 3.11.11, 4.0-rc1, 4.0
• Component/s: Cluster/Schema
• Bug Category: Availability
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.23

Source Control Link

https://github.com/apache/cassandra/commit/5f09226a0d56f6d2d5e60f83465f9f17beed0572

Test and Documentation Plan

circle

• Assignee: Blake Eggleston
• Reporter: Matt Lehman

Authors

Blake Eggleston

Reviewers

Blake Eggleston, Brandon Williams

• Created: 24/May/21 23:03
• Updated: 27/Sep/21 13:53
• Resolved: 28/May/21 23:45

Description

 

After CASSANDRA-15158 operators are no longer permitted to replace a terminated node with a stale schema. That is, launching a node to replace NodeA in the following scenario:

NodeA (terminated): schema=V0
All others (alive): schema=V1

yields:

ERROR [main] 2021-04-30 19:10:23,410 CassandraDaemon.java:822 - Exception encountered during startup
java.lang.RuntimeException: Didn't receive schemas for all known versions within the timeout
        at org.apache.cassandra.service.StorageService.waitForSchema(StorageService.java:887) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:937) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:746) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:654) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:374) [nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:616) [nf-cassandra-3.0.24.1.jar:3.0.24.1]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:809) [nf-cassandra-3.0.24.1.jar:3.0.24.1]

This can be reproduced like so:

1. Shut down C* on one node in a 3.0.24 cluster
2. Create a new keyspace and table from one of the other nodes
3. Terminate and replace the node on which C* was shut down.

Waiting for agreement of the nodes not being replaced seems prudent as not doing so could induce data loss, the node being replaced should be exempted from this check.

Reference CASSANDRA-16577 for more context.

Issue Links

• is duplicated by: CASSANDRA-16998 replace_address does not work in 3.11.10