[OAK-3349] Partial compaction

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.5, 1.8.0
• Component/s: segment-tar
• Labels: compaction gc scalability
• Epic Link: TarMK GC scalability
• Assignee: Francesco Mari
• Reporter: Michael Dürig
• Created: 04/Sep/15 08:45
• Updated: 15/Nov/23 17:11
• Resolved: 27/Jul/17 09:09

Description

On big repositories compaction can take quite a while to run as it needs to create a full deep copy of the current root node state. For such cases it could be beneficial if we could partially compact the repository thus splitting full compaction over multiple cycles.
Partial compaction would run compaction on a sub-tree just like we now run it on the full tree. Afterwards it would create a new root node state by referencing the previous root node state replacing said sub-tree with the compacted one.

Todo: Asses feasibility and impact, implement prototype.

Issue Links

• blocks: OAK-6399 Re-establish full offline compaction functionality
• depends upon: OAK-6009 Simplify cancellation of compaction by timeout
• depends upon: OAK-5996 Revisions.setHead(Function) should return the new head or null instead of boolean
• depends upon: OAK-6005 Add record id of the compacted root to the GC journal
• is blocked by: OAK-4122 Replace the commit semaphore in the segment node store with a scheduler
• is related to: OAK-6456 Include the tail generation in the TAR index
• is related to: OAK-6457 Increment the segment version number
• is related to: OAK-6458 Increment the manifest version number
• is related to: OAK-6468 Include the tail generation in the binary references index
• is related to: OAK-6484 Include tail generation in the GC journal
• is related to: OAK-6485 Expose tail and full compaction via different methods
• relates to: OAK-6520 Improve tail compactions resilience when base state cannot be determined
• relates to: OAK-6519 Properly handle tail compactions in deduplication caches
• relates to: OAK-6522 Implement unit tests for OnlineCompactor
• links to: GitHub Pull Request #773
• links to: GitHub Pull Request #1216