[OAK-4528] diff calculation in DocumentNodeStore should try to re-use journal info on diff cache miss

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.8, 1.5.6, 1.6.0
• Component/s: core, documentmk
• Labels: observation resilience
• Epic Link: Improve observation resilience
• Assignee: Marcel Reutegger
• Reporter: Vikas Saurabh
• Created: 01/Jul/16 14:12
• Updated: 01/Sep/17 05:23
• Resolved: 19/Jul/16 11:25

Description

Currently, diff information is filled into caches actively (local commits pushed in local_diff, externally read changes pushed into memory_diff). At the time of event processing though, the entries could have already been evicted.
In that case, we fall back to computing diff by comparing 2 node-states which becomes more and more expensive (and eventually fairly non-recoverable leading to OAK-2683).

To improve the situation somewhat, we can probably try to consult journal entries to read a smaller-superset of changed paths before falling down to comparison.

/cc mreutegg, chetanm, egli

Issue Links

• breaks: OAK-6495 Fallback to classic diff when journal is broken
• depends upon: OAK-4552 Checkpoints.getOldestRevisionToKeep() returns newest valid revision
• depends upon: OAK-4553 Retain journal entries for valid checkpoints
• depends upon: OAK-4556 Track branch on newBranchCommit()
• is related to: OAK-4682 ConcurrentModificationException in JournalEntry.TreeNode
• is related to: OAK-4937 JournalGC failing with RDB DocumentStore
• is related to: OAK-5228 Incorrect minTimestamp in DocumentNodeStore.diffImpl()
• is related to: OAK-2836 Create diff cache entry for merged persisted branch
• is related to: OAK-4715 Reduce DocumentStore reads for local changes
• relates to: OAK-2683 the "hitting the observation queue limit" problem