Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
None
-
Normal
Description
To have the arrival time of a mutation on a replica determine the reconciliation priority seems to provide for unintuitive database behaviour. It seems we should formalise our reconciliation logic in a manner that does not depend on this, and modify our internal APIs to prevent this dependency.
Take the following example, where both writes have the same timestamp:
Write X with a value A, TTL of 1s
Write Y with a value B, no TTL
If X and Y arrive on replicas in < 1s, X and Y are both live, so record Y wins the reconciliation. The value B appears in the database.
However, if X and Y arrive on replicas in > 1s, X is now (effectively) a tombstone. This wins the reconciliation race, and NO value is the result.
Note that the weirdness of this is more pronounced than it might first appear. If write X gets stuck in hints for a period on the coordinator to one replica, the value B appears in the database until the hint is replayed. So now we’re in a very uncertain state - will hints get replayed or not? If they do, the value B will disappear; if they don’t it won’t. This is despite a QUORUM of replicas ACKing both writes, and a QUORUM of readers being engaged on read; the database still changes state to the user suddenly at some arbitrary future point in time.
It seems to me that a simple solution to this, is to permit TTL’d data to always win a reconciliation against non-TTL’d data (of same timestamp), so that we are consistent across TTLs being transformed into tombstones.
4.0 seems like a good opportunity to fix this behaviour, and mention in CHANGES.txt.