[CASSANDRA-5667] Change timestamps used in CAS ballot proposals to be more resilient to clock skew

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.0 beta 1
• Component/s: Feature/Lightweight Transactions
• Labels: LWT

Environment

n/a

• Assignee: Jonathan Ellis
• Reporter: Nick Puz

Authors

Jonathan Ellis

• Created: 19/Jun/13 19:03
• Updated: 25/Oct/19 13:11
• Resolved: 04/Jul/13 17:02

Description

The current time is used to generate the timeuuid used for CAS ballots proposals with the logic that if a newer proposal exists then the current one needs to complete that and re-propose. The problem is that if a machine has clock skew and drifts into the future it will propose with a large timestamp (which will get accepted) but then subsequent proposals with lower (but correct) timestamps will not be able to proceed. This will prevent CAS write operations and also reads at serializable consistency level.

The work around is to initially propose with current time (current behavior) but if the proposal fails due to a larger existing one re-propose (after completing the existing if necessary) with the max of (currentTime, mostRecent+1, proposed+1).

Since small drift is normal between different nodes in the same datacenter this can happen even if NTP is working properly and a write hits one node and a subsequent serialized read hits another. In the case of NTP config issues (or OS bugs with time esp around DST) the unavailability window could be much larger.