[CASSANDRA-14593] Read-Repair breaks partition-level update atomicity

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Coordination
• Severity: Normal
• Assignee: Unassigned
• Reporter: Benedict Elliott Smith
• Created: 25/Jul/18 16:05
• Updated: 03/Jun/21 22:24

Description

We advertise generally that updates apply atomically within a partition, and users depend on the fact that visibility is guaranteed to coincide, i.e. that two rows updated at once will only be seen together.
 
However, a read that only touches one of these rows can lead to a quorum of nodes having only part of the update.
 
For instance, let’s take either a write at CL.ONE, or a failed write at QUORUM, that updates two rows.  The write makes it to only one node.  A read is performed that only touches one of these rows, and happens to include the node that received the write so read-repair is engaged.  Now, all nodes are updated to contain this row touched by the read-repair.
 
A new QUORUM read is performed covering both rows, and the node that received the whole update is not queried.  Now only the first row is returned.
 
Any application depending on both rows being visible together, for any query that requests them, will now be in an indeterminate state.
 
This seems like a fairly dangerous contractual failure, and one we should explicitly document, and we should permit disabling read-repair entirely for users who need it.
 
Or, perhaps preferably, we could modify read-repair to perform a whole-partition repair, that is only triggered by the detection of inconsistency (rather than being directly driven by the requested data).

Issue Links

• is related to: CASSANDRA-16710 Read repairs can break row isolation
• is related to: CASSANDRA-14635 Support table level configuration of monotonic reads