[OAK-4598] Collection of references retrieves less when large number of blobs added

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Segment Tar 0.0.10
• Component/s: segment-tar
• Labels: datastore gc
• Epic Link: Segment Tar
• Assignee: Francesco Mari
• Reporter: Amit Jain
• Created: 26/Jul/16 05:08
• Updated: 10/Oct/16 09:48
• Resolved: 19/Aug/16 09:51

Description

When large number of external blobs are added to the DataStore (50000) and a cycle of compaction executed then the reference collection logic only returns lesser number of blob references. It reports correct number of blob references when number of blobs added are less indicatingsome sort of overflow.
Another related issue observed when testing with lesser number of blobs is that the references returned are double the amount expected, so maybe there should be some sort of de-duplication which should be added.

Without compaction the blob references are returned correctly atleast till 100000 (ExternalBlobId#testNullBlobId)

Issue Links

• relates to: OAK-4603 The binary references index should keep track of the segments owning the references
• relates to: OAK-4604 The binary references index should be trimmed down when segments are garbage collected