The way we handle schema changes currently is inherently racy.
All of our SchemaAlteringStatement s perform validation on a schema state that's won't necessarily be there when the statement gets executed and mutates schema.
We should make all the *Metadata classes (KeyspaceMetadata, TableMetadata, ColumnMetadata, immutable, and local schema persistently snapshottable, with a single top-level AtomicReference to the current snapshot. Have DDL statements perform validation and transformation on the same state.
In pseudo-code, think
public interface DDLStatement
* Validates that the DDL statement can be applied to the provided schema snapshot.
* @param schema snapshot of schema before executing CREATE KEYSPACE
void validate(SchemaSnapshot schema);
* Applies the DDL statement to the provided schema snapshot.
* Implies that validate() has already been called on the provided snapshot.
* @param schema snapshot of schema before executing the statement
* @return snapshot of schema as it would be after executing the statement
SchemaSnapshot transform(SchemaSnapshot schema);