[OAK-3348] Cross gc sessions might introduce references to pre-compacted segments

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.2, Segment Tar 0.0.2, 1.6.0
• Component/s: segment-tar
• Labels: cleanup compaction gc
• Epic Link: SegmentMK revision GC
• Assignee: Michael Dürig
• Reporter: Michael Dürig
• Created: 04/Sep/15 08:32
• Updated: 30/Jun/16 11:56
• Resolved: 25/Apr/16 10:49

Description

I suspect that certain write operations during compaction can cause references from compacted segments to pre-compacted ones. This would effectively prevent the pre-compacted segments from getting evicted in subsequent cleanup phases.

The scenario is as follows:

• A session is opened and a lot of content is written to it such that the update limit is exceeded. This causes the changes to be written to disk.
• Revision gc runs causing a new, compacted root node state to be written to disk.
• The session saves its changes. This causes rebasing of its changes onto the current root (the compacted one). At this point any node that has been added will be added again in the sub-tree rooted at the current root. Such nodes however might have been written to disk before revision gc ran and might thus be contained in pre-compacted segments. As I suspect the node-add operation in the rebasing process not to create a deep copy of such nodes but to rather create a reference to them, a reference to a pre-compacted segment is introduced here.

Going forward we need to validate above hypothesis, assess its impact if necessary come up with a solution.

Issue Links

• blocks: OAK-4054 FileStore.containsSegment returns alway true (almost)
• blocks: OAK-4121 CompactionMap#get not transitive across compaction map generations
• blocks: OAK-4255 CLONE - FileStore.containsSegment returns alway true (almost)
• blocks: OAK-3668 Potential test failure: CompactionAndCleanupIT#testMixedSegments
• blocks: OAK-4243 Oak Segment Tar Module
• blocks: OAK-4015 Expedite commits from the compactor
• breaks: OAK-4307 SegmentWriter saves references to external blobs
• is blocked by: OAK-3864 Filestore cleanup removes referenced segments
• is blocked by: OAK-3550 Add meta data to segments
• is blocked by: OAK-3560 Tooling for writing segment graphs to a file
• is cloned by: OAK-4256 CLONE - Cross gc sessions might introduce references to pre-compacted segments
• is related to: OAK-4525 Unreferenced node records are not marked as root records in the segment
• is related to: OAK-4276 Refactor / rework compaction strategies
• is related to: OAK-4292 Document oak-segment-tar
• is related to: OAK-4295 Proper versioning of storage format
• relates to: OAK-4291 FileStore.flush prone to races leading to corruption
• relates to: OAK-4280 Compaction cannot be cancelled
• relates to: OAK-4284 Garbage left behind when compaction does not succeed
• relates to: OAK-4283 Align GCMonitor API with implementation
• relates to: OAK-4285 Align cleanup of segment id tables with the new cleanup strategy
• relates to: OAK-4279 Rework offline compaction
• relates to: OAK-4278 Fix backup and restore
• relates to: OAK-4277 Finalise de-duplication caches
• relates to: OAK-4281 Rework memory estimation for compaction
• relates to: OAK-4282 Make the number of retained gc generation configurable
• relates to: OAK-4286 Rework failing tests in CompactionAndCleanupIT
• relates to: OAK-4287 Disable / remove SegmentBufferWriter#checkGCGen
• relates to: OAK-4293 Refactor / rework compaction gain estimation
• relates to: OAK-4288 TarReader.calculateForwardReferences only used by oak-run graph tool
• relates to: OAK-4289 Remove the gc generation from the segment meta data
• relates to: OAK-4290 Update segment parser to work with the new segment format

Sub-Tasks

• Tool for detecting references to pre compacted segments: Closed
• Bump segment version to 12: Closed