[HDDS-6517] Snapshot support for Ozone - ASF JIRA

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.3.0
Fix Version/s: 1.4.0
Component/s: OM, Ozone Datanode, Ozone Filesystem, Ozone Manager
Labels:
None

Target Version/s:

1.4.0

Description

This Jira will be used to track snapshot feature support in Ozone. We will add/track details about requirements/Ozone Snapshot model/APIs/Design as part of this Jira.

API doc addmendum1 shared link:

API addendum1

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

(External) Snapshot Requirements (revised final draft).pdf
09/Jun/22 02:18
102 kB
Prashant Pogde
Encryption Zones Support in Snapshot Feature.pdf
09/Jun/22 18:45
185 kB
Neil Joshi
Snapshot_APIs_For_Ozone_shared_externally_v1.pdf
09/Jun/22 02:20
53 kB
Prashant Pogde
Snapshot_Diff_Handling_Renames_shared_externally_v1.pdf
09/Jun/22 02:24
98 kB
Prashant Pogde
Snapshot_Model_Ozone_shared_externally_v1.pdf
09/Jun/22 02:19
198 kB
Prashant Pogde
Snapshot_Ozone_Checkpoint_Based_Design_v1.pdf
09/Jun/22 02:22
261 kB
Prashant Pogde

Issue Links

is a parent of

HDDS-8120 Close rocksDB connection after SstFilteringTask finishes

Resolved

is part of

HDDS-6199 Support for object store Snapshots in Ozone

Resolved

Parent Feature

HDDS-7730 Ozone Snapshot Deletion & Garbage Collection

Resolved

relates to

HDDS-8544 Snapshot feature Phase 2 : Further enhancements for Ozone Snapshots

Open

Sub-Tasks

1.	OM changes for tracking Snapshot Metadata	Resolved	Neil Joshi
2.	Implement Create Snapshot API	Resolved	George Jahad
3.	API endpoint for creating a snapshot	Resolved	George Jahad
4.	Ratis transaction for creating a snapshot	Resolved	George Jahad
5.	Split out shared "Access check" code from OM	Resolved	George Jahad
6.	Split out shared "Path based access" code from OM	Resolved	George Jahad
7.	Write a diff utility for RocksDB checkpoints	Resolved	Prashant Pogde
8.	Snapshot Chain - list of snapshots per snapshottable bucket	Resolved	Neil Joshi
9.	[Snapshot] Handle RocksDB compaction DAG persistence and reconstruction	Resolved	Siyao Meng
10.	[Snapshot] Fix SnapshotInfo#dbTxSequenceNumber (de)serialization	Resolved	Siyao Meng
11.	[Snapshot] Finish TestOMSnapshotDAG, restore CompactionNode SnapshotID field for debugging	Resolved	Siyao Meng
12.	[Snapshot] Implement List Snapshot API	Resolved	Chung En Lee
13.	[Snapshot] Implement List Snapshot API Pagination	Resolved	Chung En Lee
14.	Path Based Access for Ozone Snapshots	Resolved	George Jahad
15.	Handle RocksDB compaction for Snapdiff	Resolved	Prashant Pogde
16.	Snapshot Aware OMMetadataManager	Resolved	George Jahad
17.	Implement Snap diff API	Resolved	Unassigned
18.	Rename Handling for Snapdiff	Resolved	Unassigned
19.	Make Ozone Snapshots work for FSO buckets	Resolved	George Jahad
20.	Create snapshot metadata entries	Resolved	Neil Joshi
21.	Bootstrapping slow followers/new followers	Resolved	George Jahad
22.	Background task for deleting excess snapshot data	Resolved	Sadanand Shenoy
23.	Prevent bucket from being deleted if snapshot exists	Resolved	Sadanand Shenoy
24.	[Snapshot] Implement snapshot key listing in Ozone Shell	Resolved	Christos Bisias
25.	Snapshot Create requires Double Buffer Flush thread to split the commit batch	Resolved	Hemant Kumar
26.	Ozone snapshot diff skeleton code	Resolved	Nandakumar
27.	SnapshotDiff Client Code	Resolved	Nandakumar
28.	[Snapshot] Implement `ozone fs` command for creating snapshot	Resolved	Hemant Kumar
29.	[snapshot] Add unit-testcases for Ozone snapshot path-based snapshot access feature	Resolved	Jyotirmoy Sinha
30.	Snapshot ID on followers should match the one on the OM Leader	Resolved	Hemant Kumar
31.	[snapshot] Add unit-testcases for snapshot create validation	Resolved	Jyotirmoy Sinha
32.	Expose more snapshot metrics under OMMetrics	Resolved	Christos Bisias
33.	[Snapshot] Support list snapshots using fs -ls	Resolved	Christos Bisias
34.	Compaction DAG node pruning (deleting part of the DAG > oldest retained snapshot)	Resolved	Hemant Kumar
35.	[snapshot] List Snapshot returns an empty list for a non-existent bucket	Resolved	Chung En Lee
36.	[snapshot] List Snapshot does not list snapshots created from 2nd bucket in a volume	Resolved	Chung En Lee
37.	Limit concurrent SnapshotDiff calls	Resolved	Hemant Kumar
38.	[Snapshot] SnapDiff pagination and partial diff report implementation	Resolved	Hemant Kumar
39.	[Snapshot] Skip compaction tracking when no Ozone snapshot exists	Resolved	Siyao Meng
40.	[snapshot] Add unit-testcases for Ozone Snapshot create API name validation	Resolved	Jyotirmoy Sinha
41.	Support efficient Snapdiff only until configured Snapshot history	Resolved	Swaminathan Balachandran
42.	[Snapshot] Use SST file list output from compaction DAG as SnapshotDiff input	Resolved	Siyao Meng
43.	[Snapshot] Mark all new proto message fields as optional	Resolved	Siyao Meng
44.	Snapdiff showing incorrect result.	Resolved	Sadanand Shenoy
45.	Support Snapdiff between a snapshot and active DB	Resolved	Sadanand Shenoy
46.	[snapshot] SnapshotDiff command lists unmodified file as modified	Resolved	Sadanand Shenoy
47.	[snapshot] Add unit-testcases for Ozone Snapshot List API	Resolved	Jyotirmoy Sinha
48.	SnapshotChainManager fails to successfully load snapshots on restart.	Resolved	Unassigned
49.	[snapshot] Add unit-testcases for Ozone fs createSnapshot	Resolved	Jyotirmoy Sinha
50.	[Snapshot] Handle OzoneManager#getKeyInfo with OmMetadataReader	Resolved	George Jahad
51.	[Snapshot] Improve error handling in SnapDiff and others	Resolved	Unassigned
52.	[Snapshot] Add acceptance tests	Resolved	Aswin Shakil
53.	If Snapdiff is too large to fit in memory, we should have persistence for snapdiff	Resolved	Hemant Kumar
54.	[snapshot] Add unit-testcases for Ozone SnapshotDiff	Resolved	Jyotirmoy Sinha
55.	Fix getKeyInfo API for snapshot paths.	Resolved	George Jahad
56.	[snapshot] Add unit-testcases for snapshot restore	Resolved	Jyotirmoy Sinha
57.	[Snapshot] Add unit-testcases for fs delete of bucket having snapshots	Resolved	Jyotirmoy Sinha
58.	[Snapshot] Wait for RocksDB checkpoint directory creation	Resolved	Hemant Kumar
59.	Snapdiff should only return modifications done to the bucket.	Resolved	Sadanand Shenoy
60.	[Snapshot] Prune backup SST files that can be expanded periodically	Resolved	Hemant Kumar
61.	Failures in TestOzoneSnapshotRestore	Resolved	Hemant Kumar
62.	[Snapshot] Flaky snapshot restore integration test.	Resolved	Hemant Kumar
63.	Add continuation token, page size and diff response to support Snapdiff pagination	Resolved	Hemant Kumar
64.	Server side changes to add wait time and job status for Snapdiff API	Resolved	Hemant Kumar
65.	[Snapshot] Implement Native Acl Integration tests for Snapshotting	Resolved	Mladjan Gadzic
66.	JNI for RocksDB SST Dump tool	Resolved	Swaminathan Balachandran
67.	[snapshot] Add testcase to handle snapshot name minimum-length	Resolved	Jyotirmoy Sinha
68.	[Snapshot] Add a smoketest for Snapshot ACLs	Resolved	Mladjan Gadzic
69.	[Snapshot] Add SnapDiff client option to force a full diff	Resolved	JiaChi Wang
70.	[snapshot] Add unit test cases for snapshot delete	Resolved	Jyotirmoy Sinha
71.	[Snapshot] Disable RocksDB column family auto compaction in SstFilteringService	Resolved	Swaminathan Balachandran
72.	[Snapshot] SstFilteringService to clean up snapshot's deletedTable and deletedDirTable as well	Resolved	Unassigned
73.	[Snapshot] Compaction DAG reconstruction logic potentially crashing OM on startup	Resolved	Hemant Kumar
74.	[Snapshot] Turn off snapshot DAG tracking when no snapshot exists	Resolved	Unassigned
75.	[snapshot] Snapshot Diff timeout	Resolved	Hemant Kumar
76.	SnapshotDiff command has error when differences between snapshots more than 1M keys	Resolved	Hemant Kumar
77.	[Snapshot] Block FS API access to deleted (non-active) snapshots	Resolved	Siyao Meng
78.	loadDb() used by SstFiltering service creates extraneous directories	Resolved	George Jahad
79.	[Snapshot] cancelAllBackgroundWork and close listeners in RDBStore#close	Resolved	Siyao Meng
80.	[Snapshot] Intermittent DB crash in RocksDBCheckpointDiffer	Resolved	Attila Doroszlai
81.	[snapshot] OM error due to 'Error during Snapshot sst filtering'	Resolved	Dave Teng
82.	'Graph traversal level exceeded allowed maximum' when the difference b/w snapshots is more than 3.	Resolved	Hemant Kumar
83.	[Snapshot] Reduce flakiness in testSkipTrackingWithZeroSnapshot	Resolved	Siyao Meng
84.	[Snapshot] Switch RocksDB Tools JNI to release build	Resolved	Siyao Meng
85.	Parsing Bug in SSTDumptool	Resolved	Swaminathan Balachandran
86.	[Snapshot] LRU Cache entries may get evicted/closed during long running processes	Resolved	Siyao Meng
87.	Provide API for snapDiff from OFS.	Resolved	Sadanand Shenoy
88.	[Snapshot] Clean SnapDiff job and report table	Resolved	Hemant Kumar
89.	[Snapshot] Remove RequestValidations#createValidationRequest	Resolved	Hemant Kumar
90.	distcp support to allow snapdiff between a snapshot and activeOS bucket	Resolved	Sadanand Shenoy
91.	SnapDiff: Create various configurations for snapDiff	Resolved	Hemant Kumar
92.	SnapDiff job and compaction DAG/SST file pruning synchronization	Resolved	Hemant Kumar
93.	Make DEFAULT_THREAD_POOL_SIZE of SnapshotDiffManager.java configurable for advanced user	Resolved	Hemant Kumar
94.	Catch Exception and log it in generateSnapshotDiffReport	Resolved	Hemant Kumar
95.	[Snapshot] Enable CI native building and testing	Resolved	Siyao Meng
96.	Intermittent timeout in TestRootedOzoneFileSystem#testSnapshotDiff	Resolved	Attila Doroszlai
97.	Throw exception if the SnapshotDiff size is huge	Resolved	Hemant Kumar
98.	Fix Native JDK 11, 17 compile	Resolved	Swaminathan Balachandran
99.	Add command to list buckets which have snapshots	Resolved	Chung En Lee
100.	Synchronization between Snapshot Deletes/GC and other Snapshot jobs (read/diff)	Resolved	Hemant Kumar
101.	Add mechanism to notify threads when om double buffer flushed	Resolved	George Jahad
102.	[Snapshot] Use `java.util.UUID` for `snapshotID` in `SnapshotInfo` for more efficient storage	Resolved	Hemant Kumar
103.	Enable snapshot feature with Layout Upgrade in new version	Resolved	Dave Teng
104.	Add unit test for snapshot diff	Resolved	Swaminathan Balachandran
105.	Prevent writes to keys that start with ".snapshot"	Resolved	Dave Teng
106.	Snapshot RocksDB instance should open with auto compaction off	Resolved	Hemant Kumar
107.	Close RocksDB instance in RDBStore if RDBStore initialization fails	Resolved	Hemant Kumar
108.	Synchronize tarball creation with background processes	Resolved	George Jahad
109.	[Snapshot] Add tests for SST dump tool	Resolved	Swaminathan Balachandran
110.	[Snapshot] Explicitly test SST dump tool in SnapDiff tests	Resolved	Unassigned
111.	Unit test for Snapdiff using tombstone entries	Resolved	Swaminathan Balachandran
112.	[Snapshot] Integration test for SnapDiff when OM leader failover happens	Resolved	Hemant Kumar
113.	[Snapshot] Add a config to enable or disable Ozone snapshot feature on OM	Resolved	Siyao Meng
114.	Disable compaction log when loading metadata for snapshot	Resolved	Hemant Kumar
115.	[snapshot] Ozone fs -ls on snapshot doesn't work for OBJECT_STORE	Resolved	Swaminathan Balachandran
116.	Removed duplicate declaration of mockito-core in rocksdb-checkpoint-differ	Resolved	Hemant Kumar
117.	Intermittent failure in TestOmSnapshot.checkKey	Resolved	Unassigned
118.	Robot test for finalized upgrade duplicates non-upgrade test	Resolved	Attila Doroszlai
119.	hdds-rocks-native 1.4.0-f5e225007-SNAPSHOT artifact is broken	Resolved	Prashant Pogde
120.	[snapshot] NumSnapshotLists not reflected on using 'snapshot ls' command	Resolved	Aswin Shakil
121.	Simplify snapshot diff command name	Resolved	Attila Doroszlai
122.	Add leading zeroes on to report table to optimize get Page call of Snapdiff	Resolved	Swaminathan Balachandran
123.	[Snapshot] Limit number of open files for each snapshot RocksDB instance	Resolved	Swaminathan Balachandran
124.	[Snapshot] Ability to cancel an in-progress snapdiff job	Resolved	Christos Bisias
125.	Snapshot: List snapdiff job	Resolved	Christos Bisias
126.	Cannot reuse MiniOzoneCluster dir due to RocksDBCheckpointDiffer instance cache	Resolved	Dave Teng
127.	NumSnapshotListFails metric not reflected when snapshot listed on invalid bucket	Resolved	Hemant Kumar
128.	NumSnapshotDeleted metric not reflected after a snapshot is deleted successfully	Resolved	Hemant Kumar
129.	[snapshot] OM start failed with exception "Snapshot chain corruption"	Resolved	Hemant Kumar
130.	Handle large tarballs in bootstrapping	Resolved	George Jahad
131.	SnapDiff sends incorrect entry in case of rename & overwrite	Resolved	Hemant Kumar
132.	[snapshot] Implement 'ozone fs' command for deleting snapshots	Resolved	Aswin Shakil
133.	Intermittent failure at Delete snapshot in acceptance test	Resolved	Dave Teng
134.	Snapdiff should return complete absolute path in Diff Entry	Resolved	Swaminathan Balachandran
135.	Handling of FSO bucket in SnapshotDiff generation	Resolved	Swaminathan Balachandran
136.	[Snapshot] Document snapshot access ACL behavior	Resolved	Unassigned
137.	RocksDB memory management for reads from Snapshots	Resolved	Dave Teng
138.	[snapshot] OM process crash when trying to access contents of deleted snapshot through fs api	Resolved	Siyao Meng
139.	[Snapshot] Refactor SnapshotDiffManager#getDeltaFiles	Resolved	Hemant Kumar
140.	Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation	Resolved	Hemant Kumar
141.	SnapDiff: Add unit tests	Resolved	Hemant Kumar
142.	[snapshot] OM shuts down on trying to delete same snapshot twice before reclamation	Resolved	Hemant Kumar
143.	[Snapshot] Fallback to full diff if getDetlaFiles from compaction DAG fails	Resolved	Hemant Kumar
144.	[Snapshot] When snapshot feature is disabled, block OM startup if there are still snapshots in the system	Resolved	Siyao Meng
145.	[Snapshot] Custom SnapshotCache implementation to replace LoadingCache	Resolved	Siyao Meng
146.	Return failure message to client for a failed snapshot diff jobs	Resolved	Hemant Kumar
147.	[Snapshot] Add feature documentation	Resolved	Prashant Pogde
148.	isBlockLocationSame check should be skipped if object is not OmKeyInfo object	Resolved	Swaminathan Balachandran
149.	[Snapshot] SnapshotDiff should throw better error messages for exception handling	Resolved	Swaminathan Balachandran
150.	SnapDiff : Edge case handling Unit Tests	Resolved	Swaminathan Balachandran
151.	Limit the total size of sst files in bootstrapping tarball.	Resolved	George Jahad
152.	MiniOzoneHAClusterImpl frequently hangs when writing keys to bucket.	Resolved	Christos Bisias
153.	Snapshot Diff should return path relative to bucket root	Resolved	Swaminathan Balachandran
154.	[Snapshot] Add sequence number to all compaction log entries	Resolved	Hemant Kumar
155.	[Snapshot] SstFilteringService should be stopped before copying incremental snapshot	Resolved	Swaminathan Balachandran
156.	Ignore newer compaction log entries when bootstrapping followers	Resolved	Hemant Kumar
157.	[Snapshot] SstFilteringService should use SnapshotCache	Resolved	Swaminathan Balachandran
158.	Write a standalone tool to create a visual graph from compaction log files.	Resolved	Hemant Kumar
159.	[Snapshot] Integration test to simulate staggered snapshot creation and deletion	Resolved	Dave Teng
160.	[Snapshot] Add more integration tests and acceptance tests	Resolved	JiaChi Wang
161.	[snapshot] SnapshotDiff should throw proper responses in case there are no differences to be listed	Resolved	Hemant Kumar
162.	Stress test bootstrapping with thousands of snapshots and millions of keys.	Resolved	Mladjan Gadzic
163.	[Snapshot] Check snapshot feature code coverage	Closed	Unassigned
164.	snapDiff doesn't return correct results after leadership transfer	Resolved	Hemant Kumar
165.	SSTFiltering service deleting wrong SST files from snapshot's DB dir	Resolved	Hemant Kumar
166.	Make snapshot dir wait poll max timeout configurable	Resolved	Swaminathan Balachandran
167.	Compaction DAG pruning service is not running after bootstrapping	Resolved	Hemant Kumar
168.	Integration test for data integrity test for SST filtering service run	Resolved	Hemant Kumar
169.	Add SSTFiltered flag in SnapshotInfo table.	Resolved	Swaminathan Balachandran
170.	[Snapshot] SnapDiff to use tombstone entries in SST files	Resolved	Swaminathan Balachandran
171.	Reduce time of compaction pause during bootstrapping	Resolved	George Jahad
172.	Link rocksdb lib to Ozone rocksdb tools lib relative path instead of absolute path	Resolved	Swaminathan Balachandran
173.	Reclaimable Filter for Snapshots garbage reclaimation	Open	Unassigned

Activity

People

Assignee:: Prashant Pogde

Reporter:: Prashant Pogde

Votes:: 0 Vote for this issue

Watchers:: 10 Start watching this issue

Dates

Created:: 26/Mar/22 20:56

Updated:: 21/Jan/24 15:29

Resolved:: 20/Jul/23 19:14