[OAK-3573] BackgroundObserver should decide about collapsing external events based on backend

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: core
• Assignee: Unassigned
• Reporter: Vikas Saurabh
• Created: 30/Oct/15 11:12
• Updated: 10/Jan/17 11:45

Description

This is a follow up of OAK-3372. With OAK-3372, BackgroundObserver by default won't collapse external events if observation queue is not full. Although, it exposes a system property to control the behavior, but it's better if this is done based on persistence used in the back-end.

The reason to still have the collapse is because of this comment

I assume whenever cluster support comes it there'd some way where external changes are polled for and kept.

Probably yes but I don't want to speculate too much here as there are currently no plans to that extend. However, the way segment node store works it might be favourable to have this collapsing in place anyway. So I'd rather keep it but add an option to disable it if the back end chooses to. Maybe the commit info could be used to convey that information in a backward compatible way?

A couple of approaches to do it were mentioned here

• Background observer can be passed the configuration to collapse/not-collapse external events early using descriptors
• Since external event can't have non-null commit info, we might hack around and add some extra data into local events - that extra info essentially can configure the background observer. Apart from this being very hacky, it's also hard to generate solve the case when there are no local events getting generated

Issue Links

• relates to: OAK-3372 Collapsing external events in BackgroundObserver even before queue is full leads to JournalEntry not getting used
• relates to: OAK-4808 Index external changes as part of NRT indexing