[OAK-6702] Improve cold standby resiliency to incoherent configs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: segment-tar, tarmk-standby
• Labels: cold-standby resilience
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 22/Sep/17 09:11
• Updated: 29/Jan/18 15:32

Description

In order to correctly configure cold standby there are two OSGi configurations that need to be provided. Among other settings, org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config needs standby=B"true" and org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config needs mode="standby". The problem is that sometimes we have mode="standby" in StandbyStoreService and standby=B"false" in SegmentNodeStoreService which leads to starting a problematic standby instance (with primary behaviour enabled, e.g. indexing, etc.). This problem stems from the fact that there are two components whose configuration should be coordinated. Proposals to mitigate this:

1. Keep the mode="standby", but merge the configuration of StandbyStoreService in the one for SegmentNodeStoreService and eliminate StandbyStoreService altogether
2. StandbyStoreService should derive mode="standby" from "standby=B"true" in SegmentNodeStoreService
3. SegmentNodeStoreService should derive "standby=B"true" from mode="standby" in StandbyStoreService even if this is backwards when compared to how the synchronization currently happens, with StandbyStoreService waiting for for a proper initialisation of SegmentNodeStoreService
4. Make StandbyStoreService configuration mandatory, but require a mode="off" setting. This way the removal of standby=B"true" from SegmentNodeStoreService would be guaranteed and any synchronization between the two components would be avoided.

/cc frm, volteanu, mduerig