[OAK-3976] journal should support large(r) entries

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.14
• Fix Version/s: 1.5.16, 1.6.0
• Component/s: documentmk
• Assignee: Vikas Saurabh
• Reporter: Stefan Egli
• Created: 03/Feb/16 10:36
• Updated: 20/Jun/18 14:43
• Resolved: 13/Dec/16 20:15

Description

Journal entries are created in the background write. Normally this happens every second. If for some reason there is a large delay between two background writes, the number of pending changes can also accumulate. Which can result in (arbitrary) large single journal entries (ie with large _c property).

This can cause multiple problems down the road:

• journal gc at this point loads 450 entries - and if some are large this can result in a very large memory consumption during gc (which can cause severe stability problems for the VM, if not OOM etc). This should be fixed with OAK-3001 (where we only get the id, thus do not care how big _c is)
• before OAK-3001 is done (which is currently scheduled after 1.4) what we can do is reduce the delete batch size (OAK-3975)
• background reads however also read the journal entries and even if OAK-3001/OAK-3975 are implemented the background read can still cause large memory consumption. So we need to improve this one way or another.

Issue Links

• is related to: OAK-7564 Commit fails when forced journal push throws exception
• is related to: OAK-3001 Simplify JournalGarbageCollector using a dedicated timestamp property
• is related to: OAK-3975 Change DELETE_BATCH_SIZE to 100 (from 450), and make it configurable
• is related to: OAK-6479 misleading log entry when there's no revision to write to the journal