[CASSANDRA-16856] Prevent broken concurrent schema pulls

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.12, 4.0.1, 4.1-alpha1, 4.1
• Component/s: Cluster/Gossip
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.11.x

Source Control Link

https://github.com/apache/cassandra/commit/23b61a5fa1de17cc6b8a1d7c300053160bfc728a

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Berenguer Blasi

Authors

Berenguer Blasi

Reviewers

Brandon Williams

• Created: 16/Aug/21 05:27
• Updated: 27/May/22 19:25
• Resolved: 19/Aug/21 05:45

Description

There's a race condition around pulling schema changes, that can occur in case the schema changes push/propagation mechanism is not immediately effective (e.g. because of network delay, or because of the pulling node being down, etc.).

If schema changes happen on node 1, these changes do not reach node 2 immediately through the SCHEMA.PUSH mechanism, and are first recognized during gossiping, the corresponding SCHEMA.PULL request from node 2 can catch the node 1 schema in the middle of it being modified by another schema change request. This can easily lead to problems (e.g. if a new table is being added, and the node 2 request reads the changes that need to be applied to system_schema.tables, but not the ones that need to be applied to system_schema.columns).

This PR addresses that by synchronizing the SCHEMA.PULL "RPC call" executed in node 1 by a request from node 2 with the method for applying schema changes in node 1.

Issue Links

• is a parent of: CASSANDRA-16996 Prevent broken concurrent schema read/writes
• links to: CASSANDRA-16856 PR 3.11
• links to: CASSANDRA-16856 PR 4.0
• links to: CASSANDRA-16856 PR trunk