[OAK-6110] Offline compaction uses too much memory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0, 1.8.0
• Component/s: segment-tar
• Labels: memory performance
• Epic Link: TarMK GC scalability
• Assignee: Michael Dürig
• Reporter: Michael Dürig
• Created: 20/Apr/17 11:36
• Updated: 12/Dec/17 13:41
• Resolved: 20/Apr/17 12:35

Description

Using offline compaction on a repository with nodes having many direct child node I observed a steady increase of heap usage. This is cause by using a MemoryNodeBuilder in CompactDiff.childNodeAdded(), which causes all those child nodes to be cached in memory.

Changing the line

child = EMPTY_NODE.builder();

to

child = writer.writeNode(EMPTY_NODE).builder();

fixes the problem as the latter returns a SegmentNodeBuilder where the former returns a MemoryNodeBuilder.

Issue Links

• is related to: OAK-6560 Sidegrade uses too much memory
• relates to: OAK-4949 SegmentWriter buffers child node list changes
• relates to: OAK-6642 Backport OAK-6110 to 1.6 (Offline compaction uses too much memory)