Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16692

Unable to replace node with stale schema

    XMLWordPrintableJSON

    Details

      Description

       

      After CASSANDRA-15158 operators are no longer permitted to replace a terminated node with a stale schema. That is, launching a node to replace NodeA in the following scenario:

      NodeA (terminated): schema=V0
      All others (alive): schema=V1

      yields:

      ERROR [main] 2021-04-30 19:10:23,410 CassandraDaemon.java:822 - Exception encountered during startup
      java.lang.RuntimeException: Didn't receive schemas for all known versions within the timeout
              at org.apache.cassandra.service.StorageService.waitForSchema(StorageService.java:887) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:937) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:746) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.StorageService.initServer(StorageService.java:654) ~[nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:374) [nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:616) [nf-cassandra-3.0.24.1.jar:3.0.24.1]
              at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:809) [nf-cassandra-3.0.24.1.jar:3.0.24.1]

      This can be reproduced like so:

      1. Shut down C* on one node in a 3.0.24 cluster
      2. Create a new keyspace and table from one of the other nodes
      3. Terminate and replace the node on which C* was shut down.

      Waiting for agreement of the nodes not being replaced seems prudent as not doing so could induce data loss, the node being replaced should be exempted from this check.

      Reference CASSANDRA-16577 for more context.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bdeggleston Blake Eggleston
                Reporter:
                mattl Matt Lehman
                Authors:
                Blake Eggleston
                Reviewers:
                Blake Eggleston, Brandon Williams
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: