When a node begins bootstrapping, Cassandra recalculates pending tokens for each keyspace that exists when the state change is observed (in StorageService:handleState*). When new keyspaces are created, we do not recalculate pending ranges (around Schema:merge). As a result, writes for new keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When bootstrapping finishes, the node which just bootstrapped will not have data for the newly created keyspace.
Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and when it finishes bootstrapping, C will cede ownership of some ranges to D. A quorum write is acknowledged by C and A. B missed the write, and the coordinator didn't send it to D at all. When D finishes bootstrapping, the quorum B+D will not contain the mutation.
Steps to reproduce:
- Join a node in BOOT mode
- Create a keyspace
- Send writes to that keyspace
- On the joining node, observe that nodetool cfstats records zero writes to the new keyspace
I have observed this directly in Cassandra 3.0, and based on my reading the code, I believe it affects up through trunk.