[OAK-1453] MongoMK failover support for replica sets (esp. shards)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: mongomk
• Labels: production resilience
• Epic Link: DocumentMK resilience
• Assignee: Unassigned
• Reporter: Michael Marth
• Created: 21/Feb/14 13:37
• Updated: 17/Jul/18 11:51
• Resolved: 09/Feb/16 07:06

Description

With OAK-759 we have introduced replica support in MongoMK. I think we still need to address the resilience for failover from primary to secoandary:
Consider a case where Oak writes to the primary. Replication to secondary is ongoing. During that period the primary goes down and the secondary becomes primary. There could be some "half-replicated" MVCC revisions, which need to be either discarded or be ignored after the failover.
This might not be an issue if there is only one shard, as the commit root is written last (and replicated last)
But with 2 shards the the replication state of these 2 shards could be inconsistent. Oak needs to handle such a situation without falling over.
If we can detect a Mongo failover we could query Mongo which revisions are fully replicated to the new primary and discard the potentially half-replicated revisions.

Issue Links

• is blocked by: OAK-1650 NPE and MicroKernelException: The node .. does not exist, on replica primary crash during save
• is blocked by: OAK-1641 Mongo: Un-/CheckedExecutionException on replica-primary crash
• is blocked by: OAK-1649 NamespaceException: OakNamespace0005 on save, after replica crash
• relates to: OAK-2592 Commit does not ensure w:majority
• relates to: OAK-3554 Use write concern of w:majority when connected to a replica set