[SOLR-7347] clock skew can cause data loss

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SolrCloud
• Assignee: Unassigned
• Reporter: Yonik Seeley
• Created: 03/Apr/15 15:32
• Updated: 03/Apr/15 15:36

Description

The high bits of versions are created using the system clock.
System clock skew on the order of magnitude of time it takes for one leader to receive it's last update to the time it takes another replica to become a leader can cause data loss for any updates to the same document until the new leaders clock catches up with the old leaders clock.

1) replica1 is the leader and indexes document A, choosing version X (and forwards to replicas)
2) replica1 goes down
3) replica2 becomes the new leader
4) replica2 indexes an update for document A, and chooses version Y (which is less than X due to clock skew) and forwards to replica3
5) replica3 checks for reordered updates, finds version X and thus drops version Y

This should be rare... you need a big enough clock skew and updates to the same document with different leaders within that time window. We should still fix this of course.

Issue Links

• relates to: SOLR-7332 Seed version buckets with max version from index