[HDDS-7743] [Snapshot] Implement snapshot disk usage

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Labels: pull-request-available
• Assignee: Aswin Shakil
• Reporter: Siyao Meng
• Created: 06/Jan/23 20:03
• Updated: 10/Oct/23 20:44
• Resolved: 05/Oct/23 18:08

Description

(Excerpt from the design doc)

We could show two sizes of a snapshot (terms borrowed from btrfs):

a. Referenced size: total size of all the keys in the snapshot. This indicates how much space it would take on the target cluster when replicated. Basically this is du on the bucket inside the DB checkpoint. Use the existing counter (space used) implemented for quota.

b. Exclusive size: the size of keys exclusively referenced by this snapshot. This indicates how much space (times replication factor) can be released from the cluster when the snapshot is deleted and garbage collected.

The exclusive size calculation can be done with the help of deletedTable in the next snapshot (or active filesystem’s if it is the last snapshot). Iterate over next snapshot’s deletedTable (and delayedDeletedTable), count the number of keys that exist in the current snapshot, and return the sum of those key sizes.

(ii) The exclusive size of a snapshot can fluctuate (typically increase) when a neighboring snapshot is deleted because of the deletedTable merge. Still, for a read-only snapshot, we can calculate this in a background thread and persist the size. And whenever a neighboring snapshot is deleted, the persisted value is invalidated and would have to be calculated again (possibly incrementally).

Issue Links

• Dependency: HDDS-9188 Add pre-replication bucket usage counter
• is a parent of: HDDS-9023 Intermittent failure in TestSnapshotDeletingService#testSnapshotWithFSO
• is a parent of: HDDS-9159 [Snapshot] Implement snapshot disk usage: Referenced size
• is a parent of: HDDS-9312 Expand all the entries deletedDirTable for Snapshot
• is a parent of: HDDS-9426 Calculate Exclusive size for deep cleaned snapshot's deleted directories
• links to: GitHub Pull Request #5301