The blob collection phase (Identifying all the blobs available in the data store) is quite an expensive part of the whole GC process, taking up a few hours sometimes on large repositories, due to iteration of the sub-folders in the data store.
- Blobs ids are tracked when the blobs are added for e.g. in a simple file in the datastore per cluster node.
- GC then consolidates this file from all the cluster nodes and uses it to get the candidates for GC.
- This variant of the MarkSweepGC can be triggered more frequently. It would be ok to miss blob id additions to this file during a crash etc., as these blobs can be cleaned up in the regular MarkSweepGC cycles triggered occasionally.
We also may be able to track other metadata along with the blob ids like paths, timestamps etc. for auditing/analytics, in-conjunction with OAK-3140.