[OAK-3864] Filestore cleanup removes referenced segments

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.3.14, 1.4
• Component/s: segmentmk
• Labels: regression
• Epic Link: SegmentMK revision GC
• Assignee: Michael Dürig
• Reporter: Michael Dürig
• Created: 12/Jan/16 14:05
• Updated: 20/Apr/16 15:47
• Resolved: 13/Jan/16 19:27

Description

In some situations FileStore.cleanup() may remove segments that are still referenced, subsequently causing a SNFE.

This is a regression introduced with OAK-1828.

FileStore.cleanup() relies on the ordering of the segments in the tar files: later segments only reference earlier segments. As we have seen in other places this assumption does not hold any more (e.g. OAK-3794, OAK-3793) since OAK-1828.
cleanup traverses the segments backwards maintaining a list of referenced ids. When a segment is not in that list, it is removed. However, this approach does not work with forward references as those are only seen later when the segment has been removed already.

cc [~alex.parvulescu], Francesco Mari

Issue Links

• blocks: OAK-4256 CLONE - Cross gc sessions might introduce references to pre-compacted segments
• blocks: OAK-3348 Cross gc sessions might introduce references to pre-compacted segments
• relates to: OAK-3972 Forward edges missing in SegmentGraph