[OAK-4201] Add an index of binary references in a tar file

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Segment Tar 0.0.4
• Component/s: segment-tar
• Epic Link: Segment Tar
• Assignee: Francesco Mari
• Reporter: Chetan Mehrotra
• Created: 13/Apr/16 14:57
• Updated: 20/Jul/16 06:43
• Resolved: 11/Jul/16 16:04

Description

Currently for Blob GC in case of segment SegmentBlobReferenceRetriever goes through all tar files and extracts the binary references. This has 2 issues

1. Logic has go through all the segments in all tar files
2. All segments get loaded in memory once which would affect normal system performance

This process can be optimized if we also write a file entry in tar (similar to gph i.e. graph and idx i.e. index files) which has entries of all binary references referred to in any segment present in that tar file. Then GC logic would just have read this file and avoid scanning all the segments

Issue Links

• is related to: OAK-4577 BlobGC performance improvements
• relates to: OAK-4312 Fix test failures in SegmentDataStoreBlobGCIT