[HDDS-6517] Snapshot support for Ozone

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: OM, Ozone Datanode, Ozone Filesystem, Ozone Manager
• Target Version/s: 1.4.0
• Assignee: Prashant Pogde
• Reporter: Prashant Pogde
• Created: 26/Mar/22 20:56
• Updated: 21/Jan/24 15:29
• Resolved: 20/Jul/23 19:14

Description

This Jira will be used to track snapshot feature support in Ozone. We will add/track details about requirements/Ozone Snapshot model/APIs/Design as part of this Jira.

 

API doc addmendum1 shared link:

API addendum1

Issue Links

• is a parent of: HDDS-8120 Close rocksDB connection after SstFilteringTask finishes
• is part of: HDDS-6199 Support for object store Snapshots in Ozone
• Parent Feature: HDDS-7730 Ozone Snapshot Deletion & Garbage Collection
• relates to: HDDS-8544 Snapshot feature Phase 2 : Further enhancements for Ozone Snapshots

Sub-Tasks

• OM changes for tracking Snapshot Metadata: Resolved
• Implement Create Snapshot API: Resolved
• API endpoint for creating a snapshot: Resolved
• Ratis transaction for creating a snapshot: Resolved
• Split out shared "Access check" code from OM: Resolved
• Split out shared "Path based access" code from OM: Resolved
• Write a diff utility for RocksDB checkpoints: Resolved
• Snapshot Chain - list of snapshots per snapshottable bucket: Resolved
• [Snapshot] Handle RocksDB compaction DAG persistence and reconstruction: Resolved
• [Snapshot] Fix SnapshotInfo#dbTxSequenceNumber (de)serialization: Resolved
• [Snapshot] Finish TestOMSnapshotDAG, restore CompactionNode SnapshotID field for debugging: Resolved
• [Snapshot] Implement List Snapshot API: Resolved
• [Snapshot] Implement List Snapshot API Pagination: Resolved
• Path Based Access for Ozone Snapshots: Resolved
• Handle RocksDB compaction for Snapdiff: Resolved
• Snapshot Aware OMMetadataManager: Resolved
• Implement Snap diff API: Resolved
• Rename Handling for Snapdiff: Resolved
• Make Ozone Snapshots work for FSO buckets: Resolved
• Create snapshot metadata entries: Resolved
• Bootstrapping slow followers/new followers: Resolved
• Background task for deleting excess snapshot data: Resolved
• Prevent bucket from being deleted if snapshot exists: Resolved
• [Snapshot] Implement snapshot key listing in Ozone Shell: Resolved
• Snapshot Create requires Double Buffer Flush thread to split the commit batch: Resolved
• Ozone snapshot diff skeleton code: Resolved
• SnapshotDiff Client Code: Resolved
• [Snapshot] Implement `ozone fs` command for creating snapshot: Resolved
• [snapshot] Add unit-testcases for Ozone snapshot path-based snapshot access feature: Resolved
• Snapshot ID on followers should match the one on the OM Leader: Resolved
• [snapshot] Add unit-testcases for snapshot create validation: Resolved
• Expose more snapshot metrics under OMMetrics: Resolved
• [Snapshot] Support list snapshots using fs -ls: Resolved
• Compaction DAG node pruning (deleting part of the DAG > oldest retained snapshot): Resolved
• [snapshot] List Snapshot returns an empty list for a non-existent bucket: Resolved
• [snapshot] List Snapshot does not list snapshots created from 2nd bucket in a volume: Resolved
• Limit concurrent SnapshotDiff calls: Resolved
• [Snapshot] SnapDiff pagination and partial diff report implementation: Resolved
• [Snapshot] Skip compaction tracking when no Ozone snapshot exists: Resolved
• [snapshot] Add unit-testcases for Ozone Snapshot create API name validation: Resolved
• Support efficient Snapdiff only until configured Snapshot history: Resolved
• [Snapshot] Use SST file list output from compaction DAG as SnapshotDiff input: Resolved
• [Snapshot] Mark all new proto message fields as optional: Resolved
• Snapdiff showing incorrect result.: Resolved
• Support Snapdiff between a snapshot and active DB: Resolved
• [snapshot] SnapshotDiff command lists unmodified file as modified: Resolved
• [snapshot] Add unit-testcases for Ozone Snapshot List API: Resolved
• SnapshotChainManager fails to successfully load snapshots on restart.: Resolved
• [snapshot] Add unit-testcases for Ozone fs createSnapshot: Resolved
• [Snapshot] Handle OzoneManager#getKeyInfo with OmMetadataReader: Resolved
• [Snapshot] Improve error handling in SnapDiff and others: Resolved
• [Snapshot] Add acceptance tests: Resolved
• If Snapdiff is too large to fit in memory, we should have persistence for snapdiff: Resolved
• [snapshot] Add unit-testcases for Ozone SnapshotDiff: Resolved
• Fix getKeyInfo API for snapshot paths.: Resolved
• [snapshot] Add unit-testcases for snapshot restore: Resolved
• [Snapshot] Add unit-testcases for fs delete of bucket having snapshots: Resolved
• [Snapshot] Wait for RocksDB checkpoint directory creation: Resolved
• Snapdiff should only return modifications done to the bucket.: Resolved
• [Snapshot] Prune backup SST files that can be expanded periodically: Resolved
• Failures in TestOzoneSnapshotRestore: Resolved
• [Snapshot] Flaky snapshot restore integration test.: Resolved
• Add continuation token, page size and diff response to support Snapdiff pagination: Resolved
• Server side changes to add wait time and job status for Snapdiff API: Resolved
• [Snapshot] Implement Native Acl Integration tests for Snapshotting: Resolved
• JNI for RocksDB SST Dump tool: Resolved
• [snapshot] Add testcase to handle snapshot name minimum-length: Resolved
• [Snapshot] Add a smoketest for Snapshot ACLs: Resolved
• [Snapshot] Add SnapDiff client option to force a full diff: Resolved
• [snapshot] Add unit test cases for snapshot delete: Resolved
• [Snapshot] Disable RocksDB column family auto compaction in SstFilteringService: Resolved
• [Snapshot] SstFilteringService to clean up snapshot's deletedTable and deletedDirTable as well: Resolved
• [Snapshot] Compaction DAG reconstruction logic potentially crashing OM on startup: Resolved
• [Snapshot] Turn off snapshot DAG tracking when no snapshot exists: Resolved
• [snapshot] Snapshot Diff timeout: Resolved
• SnapshotDiff command has error when differences between snapshots more than 1M keys: Resolved
• [Snapshot] Block FS API access to deleted (non-active) snapshots: Resolved
• loadDb() used by SstFiltering service creates extraneous directories: Resolved
• [Snapshot] cancelAllBackgroundWork and close listeners in RDBStore#close: Resolved
• [Snapshot] Intermittent DB crash in RocksDBCheckpointDiffer: Resolved
• [snapshot] OM error due to 'Error during Snapshot sst filtering': Resolved
• 'Graph traversal level exceeded allowed maximum' when the difference b/w snapshots is more than 3.: Resolved
• [Snapshot] Reduce flakiness in testSkipTrackingWithZeroSnapshot: Resolved
• [Snapshot] Switch RocksDB Tools JNI to release build: Resolved
• Parsing Bug in SSTDumptool: Resolved
• [Snapshot] LRU Cache entries may get evicted/closed during long running processes: Resolved
• Provide API for snapDiff from OFS.: Resolved
• [Snapshot] Clean SnapDiff job and report table: Resolved
• [Snapshot] Remove RequestValidations#createValidationRequest: Resolved
• distcp support to allow snapdiff between a snapshot and activeOS bucket: Resolved
• SnapDiff: Create various configurations for snapDiff: Resolved
• SnapDiff job and compaction DAG/SST file pruning synchronization: Resolved
• Make DEFAULT_THREAD_POOL_SIZE of SnapshotDiffManager.java configurable for advanced user: Resolved
• Catch Exception and log it in generateSnapshotDiffReport: Resolved
• [Snapshot] Enable CI native building and testing: Resolved
• Intermittent timeout in TestRootedOzoneFileSystem#testSnapshotDiff: Resolved
• Throw exception if the SnapshotDiff size is huge: Resolved
• Fix Native JDK 11, 17 compile: Resolved
• Add command to list buckets which have snapshots: Resolved
• Synchronization between Snapshot Deletes/GC and other Snapshot jobs (read/diff): Resolved
• Add mechanism to notify threads when om double buffer flushed: Resolved
• [Snapshot] Use `java.util.UUID` for `snapshotID` in `SnapshotInfo` for more efficient storage: Resolved
• Enable snapshot feature with Layout Upgrade in new version: Resolved
• Add unit test for snapshot diff: Resolved
• Prevent writes to keys that start with ".snapshot": Resolved
• Snapshot RocksDB instance should open with auto compaction off: Resolved
• Close RocksDB instance in RDBStore if RDBStore initialization fails: Resolved
• Synchronize tarball creation with background processes: Resolved
• [Snapshot] Add tests for SST dump tool: Resolved
• [Snapshot] Explicitly test SST dump tool in SnapDiff tests: Resolved
• Unit test for Snapdiff using tombstone entries: Resolved
• [Snapshot] Integration test for SnapDiff when OM leader failover happens: Resolved
• [Snapshot] Add a config to enable or disable Ozone snapshot feature on OM: Resolved
• Disable compaction log when loading metadata for snapshot: Resolved
• [snapshot] Ozone fs -ls on snapshot doesn't work for OBJECT_STORE: Resolved
• Removed duplicate declaration of mockito-core in rocksdb-checkpoint-differ: Resolved
• Intermittent failure in TestOmSnapshot.checkKey: Resolved
• Robot test for finalized upgrade duplicates non-upgrade test: Resolved
• hdds-rocks-native 1.4.0-f5e225007-SNAPSHOT artifact is broken: Resolved
• [snapshot] NumSnapshotLists not reflected on using 'snapshot ls' command: Resolved
• Simplify snapshot diff command name: Resolved
• Add leading zeroes on to report table to optimize get Page call of Snapdiff: Resolved
• [Snapshot] Limit number of open files for each snapshot RocksDB instance: Resolved
• [Snapshot] Ability to cancel an in-progress snapdiff job: Resolved
• Snapshot: List snapdiff job: Resolved
• Cannot reuse MiniOzoneCluster dir due to RocksDBCheckpointDiffer instance cache: Resolved
• NumSnapshotListFails metric not reflected when snapshot listed on invalid bucket: Resolved
• NumSnapshotDeleted metric not reflected after a snapshot is deleted successfully: Resolved
• [snapshot] OM start failed with exception "Snapshot chain corruption": Resolved
• Handle large tarballs in bootstrapping: Resolved
• SnapDiff sends incorrect entry in case of rename & overwrite: Resolved
• [snapshot] Implement 'ozone fs' command for deleting snapshots: Resolved
• Intermittent failure at Delete snapshot in acceptance test: Resolved
• Snapdiff should return complete absolute path in Diff Entry: Resolved
• Handling of FSO bucket in SnapshotDiff generation: Resolved
• [Snapshot] Document snapshot access ACL behavior: Resolved
• RocksDB memory management for reads from Snapshots: Resolved
• [snapshot] OM process crash when trying to access contents of deleted snapshot through fs api: Resolved
• [Snapshot] Refactor SnapshotDiffManager#getDeltaFiles: Resolved
• Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation: Resolved
• SnapDiff: Add unit tests: Resolved
• [snapshot] OM shuts down on trying to delete same snapshot twice before reclamation: Resolved
• [Snapshot] Fallback to full diff if getDetlaFiles from compaction DAG fails: Resolved
• [Snapshot] When snapshot feature is disabled, block OM startup if there are still snapshots in the system: Resolved
• [Snapshot] Custom SnapshotCache implementation to replace LoadingCache: Resolved
• Return failure message to client for a failed snapshot diff jobs: Resolved
• [Snapshot] Add feature documentation: Resolved
• isBlockLocationSame check should be skipped if object is not OmKeyInfo object: Resolved
• [Snapshot] SnapshotDiff should throw better error messages for exception handling: Resolved
• SnapDiff : Edge case handling Unit Tests: Resolved
• Limit the total size of sst files in bootstrapping tarball.: Resolved
• MiniOzoneHAClusterImpl frequently hangs when writing keys to bucket.: Resolved
• Snapshot Diff should return path relative to bucket root: Resolved
• [Snapshot] Add sequence number to all compaction log entries: Resolved
• [Snapshot] SstFilteringService should be stopped before copying incremental snapshot: Resolved
• Ignore newer compaction log entries when bootstrapping followers: Resolved
• [Snapshot] SstFilteringService should use SnapshotCache: Resolved
• Write a standalone tool to create a visual graph from compaction log files.: Resolved
• [Snapshot] Integration test to simulate staggered snapshot creation and deletion: Resolved
• [Snapshot] Add more integration tests and acceptance tests: Resolved
• [snapshot] SnapshotDiff should throw proper responses in case there are no differences to be listed: Resolved
• Stress test bootstrapping with thousands of snapshots and millions of keys.: Resolved
• [Snapshot] Check snapshot feature code coverage: Closed
• snapDiff doesn't return correct results after leadership transfer: Resolved
• SSTFiltering service deleting wrong SST files from snapshot's DB dir: Resolved
• Make snapshot dir wait poll max timeout configurable: Resolved
• Compaction DAG pruning service is not running after bootstrapping: Resolved
• Integration test for data integrity test for SST filtering service run: Resolved
• Add SSTFiltered flag in SnapshotInfo table.: Resolved
• [Snapshot] SnapDiff to use tombstone entries in SST files: Resolved
• Reduce time of compaction pause during bootstrapping: Resolved
• Link rocksdb lib to Ozone rocksdb tools lib relative path instead of absolute path: Resolved
• Reclaimable Filter for Snapshots garbage reclaimation: Open