[OAK-4949] SegmentWriter buffers child node list changes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Component/s: segment-tar
• Labels: performance
• Epic Link: TarMK GC scalability
• Assignee: Michael Dürig
• Reporter: Alex Deparvu
• Created: 18/Oct/16 10:27
• Updated: 20/Apr/17 13:09
• Resolved: 20/Apr/17 12:42

Description

The SegmentWriter currently buffers the list of child nodes changed on a nodestate update [0] (new node or updated node). This can be problematic in a scenario where there are a large number of children added to a node (ie. unique index size seen to spike above 10MM in one case).

To have a reference for the impact of this, at the SegmentWriter level, for a list of map entries of almost 3MM items, I saw it take up around 245MB heap.

This issue serves to track a possible improvement here in how we handle this update scenario.

[0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java#L516

Issue Links

• is related to: OAK-6106 Excessive memory usage by the cached segment references
• is related to: OAK-6110 Offline compaction uses too much memory