[CASSANDRA-19436] When transitioning to Accord migration it's not safe to read immediately using Accord due to concurrent non-serial writes

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.x
• Component/s: Accord, Consistency/Coordination
• Epic Link: CEP-15: Accord Beta
• Bug Category: Correctness
• Severity: Critical
• Complexity: Normal
• Discovered By: Code Inspection
• Platform: All
• Since Version: 5.x

Source Control Link

https://github.com/apache/cassandra/commit/e6bd89ac36c29f1a7d5f7bcc0167c5c56e153fdd

Test and Documentation Plan

Updated existing tests

• Assignee: Ariel Weisberg
• Reporter: Ariel Weisberg

Authors

Ariel Weisberg

Reviewers

Ariel Weisberg, Benedict Elliott Smith

• Created: 26/Feb/24 19:17
• Updated: 13/Nov/24 16:31
• Resolved: 13/Nov/24 16:31

Description

Concurrent writes at the same time that migration starts make it unsafe to read from Accord because txn recovery will not be deterministic in the presences of writes not done through Accord.

Migration to Accord needs to be split into two phases, in the first phase we write through Accord and always respect the consistency level and do synchronous commit. This allows Paxos and non-serial writes to continue while Accord's metadata covers everything needed for future reads. Paxos continues to operate as normal since it has enough metadata to allow key migration in the second phase and we need to stay online so something needs to handle LWTs.

Then repair runs and makes it possible for Accord to read any data written non-transactionally and we can then do key migration and route all updates (conditional or blind) through Accord while Paxos repair runs so we can stop doing key migration.

Issue Links

• is related to: CASSANDRA-19440 Non-serial writes can race with Accord topology changes
• is related to: CASSANDRA-19439 Non-serial reads need to handle racing with Accord topology changes
• is related to: CASSANDRA-19441 Blocking read repair needs to handle racing with Accord topology changes
• relates to: CASSANDRA-19430 Read repair through Accord needs to only route the read repair through Accord if the range is actually migrated/running on Accord
• relates to: CASSANDRA-19431 Mutations need to split Accord/non-Accord mutations based on whether migration is completed
• links to: Cassandra PR
• links to: GitHub Pull Request #3462