[OAK-3002] Optimize docCache and docChildrenCache invalidation by filtering using journal

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.19, 1.2.3, 1.3.2, 1.4
• Component/s: core, mongomk
• Labels: scalability
• Assignee: Marcel Reutegger
• Reporter: Stefan Egli
• Created: 17/Jun/15 07:00
• Updated: 02/Mar/16 11:46
• Resolved: 29/Jun/15 11:00

Description

This subtask is about spawning out a comment on OAK-2829 re optimizing docCache invalidation using the newly introduced external diff journal:

Attached OAK-2829-improved-doc-cache-invaliation.patch which is a suggestion on how to avoid invalidating the entire document cache when doing a backgroundRead but instead making use of the new journal: ie only invalidate from the document cache what has actually changed.

I'd like to get an opinion (mreutegg, chetanm?) on this first, I have a load test pending locally which found invalidation of the document cache to be the slowest part thus wanted to optimize this first.

Open still/next:

• also invalidate only necessary parts from the docChildrenCache
• junits for all of these

Issue Links

• is part of: OAK-3187 Backporting journal to 1.0 branch
• is related to: OAK-2792 Investigate feasibility/advantages of log-tailing mongo op-log to invalidate cache
• is related to: OAK-2793 Time limit for HierarchicalInvalidator
• is related to: OAK-2187 Document cache invalidation logic generating lots of garbage
• is related to: OAK-2646 Look for alternative cache invalidation logic for better performance