Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-6517

Snapshot support for Ozone

    XMLWordPrintableJSON

Details

    Description

      This Jira will be used to track snapshot feature support in Ozone. We will add/track details about requirements/Ozone Snapshot model/APIs/Design as part of this Jira.

       

      API doc addmendum1 shared link:

      API addendum1

      Attachments

        Issue Links

          1.
          OM changes for tracking Snapshot Metadata Sub-task Resolved Neil Joshi
          2.
          Implement Create Snapshot API Sub-task Resolved George Jahad
          3.
          API endpoint for creating a snapshot Sub-task Resolved George Jahad
          4.
          Ratis transaction for creating a snapshot Sub-task Resolved George Jahad
          5.
          Split out shared "Access check" code from OM Sub-task Resolved George Jahad
          6.
          Split out shared "Path based access" code from OM Sub-task Resolved George Jahad
          7.
          Write a diff utility for RocksDB checkpoints Sub-task Resolved Prashant Pogde
          8.
          Snapshot Chain - list of snapshots per snapshottable bucket Sub-task Resolved Neil Joshi
          9.
          [Snapshot] Handle RocksDB compaction DAG persistence and reconstruction Sub-task Resolved Siyao Meng
          10.
          [Snapshot] Fix SnapshotInfo#dbTxSequenceNumber (de)serialization Sub-task Resolved Siyao Meng
          11.
          [Snapshot] Finish TestOMSnapshotDAG, restore CompactionNode SnapshotID field for debugging Sub-task Resolved Siyao Meng
          12.
          [Snapshot] Implement List Snapshot API Sub-task Resolved Chung En Lee
          13.
          [Snapshot] Implement List Snapshot API Pagination Sub-task Resolved Chung En Lee
          14.
          Path Based Access for Ozone Snapshots Sub-task Resolved George Jahad
          15.
          Handle RocksDB compaction for Snapdiff Sub-task Resolved Prashant Pogde
          16.
          Snapshot Aware OMMetadataManager Sub-task Resolved George Jahad
          17.
          Implement Snap diff API Sub-task Resolved Unassigned
          18.
          Rename Handling for Snapdiff Sub-task Resolved Unassigned
          19.
          Make Ozone Snapshots work for FSO buckets Sub-task Resolved George Jahad
          20.
          Create snapshot metadata entries Sub-task Resolved Neil Joshi
          21.
          Bootstrapping slow followers/new followers Sub-task Resolved George Jahad
          22.
          Background task for deleting excess snapshot data Sub-task Resolved Sadanand Shenoy
          23.
          Prevent bucket from being deleted if snapshot exists Sub-task Resolved Sadanand Shenoy
          24.
          [Snapshot] Implement snapshot key listing in Ozone Shell Sub-task Resolved Christos Bisias
          25.
          Snapshot Create requires Double Buffer Flush thread to split the commit batch Sub-task Resolved Hemant Kumar
          26.
          Ozone snapshot diff skeleton code Sub-task Resolved Nandakumar
          27.
          SnapshotDiff Client Code Sub-task Resolved Nandakumar
          28.
          [Snapshot] Implement `ozone fs` command for creating snapshot Sub-task Resolved Hemant Kumar
          29.
          [snapshot] Add unit-testcases for Ozone snapshot path-based snapshot access feature Sub-task Resolved Jyotirmoy Sinha
          30.
          Snapshot ID on followers should match the one on the OM Leader Sub-task Resolved Hemant Kumar
          31.
          [snapshot] Add unit-testcases for snapshot create validation Sub-task Resolved Jyotirmoy Sinha
          32.
          Expose more snapshot metrics under OMMetrics Sub-task Resolved Christos Bisias
          33.
          [Snapshot] Support list snapshots using fs -ls Sub-task Resolved Christos Bisias
          34.
          Compaction DAG node pruning (deleting part of the DAG > oldest retained snapshot) Sub-task Resolved Hemant Kumar
          35.
          [snapshot] List Snapshot returns an empty list for a non-existent bucket Sub-task Resolved Chung En Lee
          36.
          [snapshot] List Snapshot does not list snapshots created from 2nd bucket in a volume Sub-task Resolved Chung En Lee
          37.
          Limit concurrent SnapshotDiff calls Sub-task Resolved Hemant Kumar
          38.
          [Snapshot] SnapDiff pagination and partial diff report implementation Sub-task Resolved Hemant Kumar
          39.
          [Snapshot] Skip compaction tracking when no Ozone snapshot exists Sub-task Resolved Siyao Meng
          40.
          [snapshot] Add unit-testcases for Ozone Snapshot create API name validation Sub-task Resolved Jyotirmoy Sinha
          41.
          Support efficient Snapdiff only until configured Snapshot history Sub-task Resolved Swaminathan Balachandran
          42.
          [Snapshot] Use SST file list output from compaction DAG as SnapshotDiff input Sub-task Resolved Siyao Meng
          43.
          [Snapshot] Mark all new proto message fields as optional Sub-task Resolved Siyao Meng
          44.
          Snapdiff showing incorrect result. Sub-task Resolved Sadanand Shenoy
          45.
          Support Snapdiff between a snapshot and active DB Sub-task Resolved Sadanand Shenoy
          46.
          [snapshot] SnapshotDiff command lists unmodified file as modified Sub-task Resolved Sadanand Shenoy
          47.
          [snapshot] Add unit-testcases for Ozone Snapshot List API Sub-task Resolved Jyotirmoy Sinha
          48.
          SnapshotChainManager fails to successfully load snapshots on restart. Sub-task Resolved Unassigned
          49.
          [snapshot] Add unit-testcases for Ozone fs createSnapshot Sub-task Resolved Jyotirmoy Sinha
          50.
          [Snapshot] Handle OzoneManager#getKeyInfo with OmMetadataReader Sub-task Resolved George Jahad
          51.
          [Snapshot] Improve error handling in SnapDiff and others Sub-task Resolved Unassigned
          52.
          [Snapshot] Add acceptance tests Sub-task Resolved Aswin Shakil
          53.
          If Snapdiff is too large to fit in memory, we should have persistence for snapdiff Sub-task Resolved Hemant Kumar
          54.
          [snapshot] Add unit-testcases for Ozone SnapshotDiff Sub-task Resolved Jyotirmoy Sinha
          55.
          Fix getKeyInfo API for snapshot paths. Sub-task Resolved George Jahad
          56.
          [snapshot] Add unit-testcases for snapshot restore Sub-task Resolved Jyotirmoy Sinha
          57.
          [Snapshot] Add unit-testcases for fs delete of bucket having snapshots Sub-task Resolved Jyotirmoy Sinha
          58.
          [Snapshot] Wait for RocksDB checkpoint directory creation Sub-task Resolved Hemant Kumar
          59.
          Snapdiff should only return modifications done to the bucket. Sub-task Resolved Sadanand Shenoy
          60.
          [Snapshot] Prune backup SST files that can be expanded periodically Sub-task Resolved Hemant Kumar
          61.
          Failures in TestOzoneSnapshotRestore Sub-task Resolved Hemant Kumar
          62.
          [Snapshot] Flaky snapshot restore integration test. Sub-task Resolved Hemant Kumar
          63.
          Add continuation token, page size and diff response to support Snapdiff pagination Sub-task Resolved Hemant Kumar
          64.
          Server side changes to add wait time and job status for Snapdiff API Sub-task Resolved Hemant Kumar
          65.
          [Snapshot] Implement Native Acl Integration tests for Snapshotting Sub-task Resolved Mladjan Gadzic
          66.
          JNI for RocksDB SST Dump tool Sub-task Resolved Swaminathan Balachandran
          67.
          [snapshot] Add testcase to handle snapshot name minimum-length Sub-task Resolved Jyotirmoy Sinha
          68.
          [Snapshot] Add a smoketest for Snapshot ACLs Sub-task Resolved Mladjan Gadzic
          69.
          [Snapshot] Add SnapDiff client option to force a full diff Sub-task Resolved JiaChi Wang
          70.
          [snapshot] Add unit test cases for snapshot delete Sub-task Resolved Jyotirmoy Sinha
          71.
          [Snapshot] Disable RocksDB column family auto compaction in SstFilteringService Sub-task Resolved Swaminathan Balachandran
          72.
          [Snapshot] SstFilteringService to clean up snapshot's deletedTable and deletedDirTable as well Sub-task Resolved Unassigned
          73.
          [Snapshot] Compaction DAG reconstruction logic potentially crashing OM on startup Sub-task Resolved Hemant Kumar
          74.
          [Snapshot] Turn off snapshot DAG tracking when no snapshot exists Sub-task Resolved Unassigned
          75.
          [snapshot] Snapshot Diff timeout Sub-task Resolved Hemant Kumar
          76.
          SnapshotDiff command has error when differences between snapshots more than 1M keys Sub-task Resolved Hemant Kumar
          77.
          [Snapshot] Block FS API access to deleted (non-active) snapshots Sub-task Resolved Siyao Meng
          78.
          loadDb() used by SstFiltering service creates extraneous directories Sub-task Resolved George Jahad
          79.
          [Snapshot] cancelAllBackgroundWork and close listeners in RDBStore#close Sub-task Resolved Siyao Meng
          80.
          [Snapshot] Intermittent DB crash in RocksDBCheckpointDiffer Sub-task Resolved Attila Doroszlai
          81.
          [snapshot] OM error due to 'Error during Snapshot sst filtering' Sub-task Resolved Dave Teng
          82.
          'Graph traversal level exceeded allowed maximum' when the difference b/w snapshots is more than 3. Sub-task Resolved Hemant Kumar
          83.
          [Snapshot] Reduce flakiness in testSkipTrackingWithZeroSnapshot Sub-task Resolved Siyao Meng
          84.
          [Snapshot] Switch RocksDB Tools JNI to release build Sub-task Resolved Siyao Meng
          85.
          Parsing Bug in SSTDumptool Sub-task Resolved Swaminathan Balachandran
          86.
          [Snapshot] LRU Cache entries may get evicted/closed during long running processes Sub-task Resolved Siyao Meng
          87.
          Provide API for snapDiff from OFS. Sub-task Resolved Sadanand Shenoy
          88.
          [Snapshot] Clean SnapDiff job and report table Sub-task Resolved Hemant Kumar
          89.
          [Snapshot] Remove RequestValidations#createValidationRequest Sub-task Resolved Hemant Kumar
          90.
          distcp support to allow snapdiff between a snapshot and activeOS bucket Sub-task Resolved Sadanand Shenoy
          91.
          SnapDiff: Create various configurations for snapDiff Sub-task Resolved Hemant Kumar
          92.
          SnapDiff job and compaction DAG/SST file pruning synchronization Sub-task Resolved Hemant Kumar
          93.
          Make DEFAULT_THREAD_POOL_SIZE of SnapshotDiffManager.java configurable for advanced user Sub-task Resolved Hemant Kumar
          94.
          Catch Exception and log it in generateSnapshotDiffReport Sub-task Resolved Hemant Kumar
          95.
          [Snapshot] Enable CI native building and testing Sub-task Resolved Siyao Meng
          96.
          Intermittent timeout in TestRootedOzoneFileSystem#testSnapshotDiff Sub-task Resolved Attila Doroszlai
          97.
          Throw exception if the SnapshotDiff size is huge Sub-task Resolved Hemant Kumar
          98.
          Fix Native JDK 11, 17 compile Sub-task Resolved Swaminathan Balachandran
          99.
          Add command to list buckets which have snapshots Sub-task Resolved Chung En Lee
          100.
          Synchronization between Snapshot Deletes/GC and other Snapshot jobs (read/diff) Sub-task Resolved Hemant Kumar
          101.
          Add mechanism to notify threads when om double buffer flushed Sub-task Resolved George Jahad
          102.
          [Snapshot] Use `java.util.UUID` for `snapshotID` in `SnapshotInfo` for more efficient storage Sub-task Resolved Hemant Kumar
          103.
          Enable snapshot feature with Layout Upgrade in new version Sub-task Resolved Dave Teng
          104.
          Add unit test for snapshot diff Sub-task Resolved Swaminathan Balachandran
          105.
          Prevent writes to keys that start with ".snapshot" Sub-task Resolved Dave Teng
          106.
          Snapshot RocksDB instance should open with auto compaction off Sub-task Resolved Hemant Kumar
          107.
          Close RocksDB instance in RDBStore if RDBStore initialization fails Sub-task Resolved Hemant Kumar
          108.
          Synchronize tarball creation with background processes Sub-task Resolved George Jahad
          109.
          [Snapshot] Add tests for SST dump tool Sub-task Resolved Swaminathan Balachandran
          110.
          [Snapshot] Explicitly test SST dump tool in SnapDiff tests Sub-task Resolved Unassigned
          111.
          Unit test for Snapdiff using tombstone entries Sub-task Resolved Swaminathan Balachandran
          112.
          [Snapshot] Integration test for SnapDiff when OM leader failover happens Sub-task Resolved Hemant Kumar
          113.
          [Snapshot] Add a config to enable or disable Ozone snapshot feature on OM Sub-task Resolved Siyao Meng
          114.
          Disable compaction log when loading metadata for snapshot Sub-task Resolved Hemant Kumar
          115.
          [snapshot] Ozone fs -ls on snapshot doesn't work for OBJECT_STORE Sub-task Resolved Swaminathan Balachandran
          116.
          Removed duplicate declaration of mockito-core in rocksdb-checkpoint-differ Sub-task Resolved Hemant Kumar
          117.
          Intermittent failure in TestOmSnapshot.checkKey Sub-task Resolved Unassigned
          118.
          Robot test for finalized upgrade duplicates non-upgrade test Sub-task Resolved Attila Doroszlai
          119.
          hdds-rocks-native 1.4.0-f5e225007-SNAPSHOT artifact is broken Sub-task Resolved Prashant Pogde
          120.
          [snapshot] NumSnapshotLists not reflected on using 'snapshot ls' command Sub-task Resolved Aswin Shakil
          121.
          Simplify snapshot diff command name Sub-task Resolved Attila Doroszlai
          122.
          Add leading zeroes on to report table to optimize get Page call of Snapdiff Sub-task Resolved Swaminathan Balachandran
          123.
          [Snapshot] Limit number of open files for each snapshot RocksDB instance Sub-task Resolved Swaminathan Balachandran
          124.
          [Snapshot] Ability to cancel an in-progress snapdiff job Sub-task Resolved Christos Bisias
          125.
          Snapshot: List snapdiff job Sub-task Resolved Christos Bisias
          126.
          Cannot reuse MiniOzoneCluster dir due to RocksDBCheckpointDiffer instance cache Sub-task Resolved Dave Teng
          127.
          NumSnapshotListFails metric not reflected when snapshot listed on invalid bucket Sub-task Resolved Hemant Kumar
          128.
          NumSnapshotDeleted metric not reflected after a snapshot is deleted successfully Sub-task Resolved Hemant Kumar
          129.
          [snapshot] OM start failed with exception "Snapshot chain corruption" Sub-task Resolved Hemant Kumar
          130.
          Handle large tarballs in bootstrapping Sub-task Resolved George Jahad
          131.
          SnapDiff sends incorrect entry in case of rename & overwrite Sub-task Resolved Hemant Kumar
          132.
          [snapshot] Implement 'ozone fs' command for deleting snapshots Sub-task Resolved Aswin Shakil
          133.
          Intermittent failure at Delete snapshot in acceptance test Sub-task Resolved Dave Teng
          134.
          Snapdiff should return complete absolute path in Diff Entry Sub-task Resolved Swaminathan Balachandran
          135.
          Handling of FSO bucket in SnapshotDiff generation Sub-task Resolved Swaminathan Balachandran
          136.
          [Snapshot] Document snapshot access ACL behavior Sub-task Resolved Unassigned
          137.
          RocksDB memory management for reads from Snapshots Sub-task Resolved Dave Teng
          138.
          [snapshot] OM process crash when trying to access contents of deleted snapshot through fs api Sub-task Resolved Siyao Meng
          139.
          [Snapshot] Refactor SnapshotDiffManager#getDeltaFiles Sub-task Resolved Hemant Kumar
          140.
          Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation Sub-task Resolved Hemant Kumar
          141.
          SnapDiff: Add unit tests Sub-task Resolved Hemant Kumar
          142.
          [snapshot] OM shuts down on trying to delete same snapshot twice before reclamation Sub-task Resolved Hemant Kumar
          143.
          [Snapshot] Fallback to full diff if getDetlaFiles from compaction DAG fails Sub-task Resolved Hemant Kumar
          144.
          [Snapshot] When snapshot feature is disabled, block OM startup if there are still snapshots in the system Sub-task Resolved Siyao Meng
          145.
          [Snapshot] Custom SnapshotCache implementation to replace LoadingCache Sub-task Resolved Siyao Meng
          146.
          Return failure message to client for a failed snapshot diff jobs Sub-task Resolved Hemant Kumar
          147.
          [Snapshot] Add feature documentation Sub-task Resolved Prashant Pogde
          148.
          isBlockLocationSame check should be skipped if object is not OmKeyInfo object Sub-task Resolved Swaminathan Balachandran
          149.
          [Snapshot] SnapshotDiff should throw better error messages for exception handling Sub-task Resolved Swaminathan Balachandran
          150.
          SnapDiff : Edge case handling Unit Tests Sub-task Resolved Swaminathan Balachandran
          151.
          Limit the total size of sst files in bootstrapping tarball. Sub-task Resolved George Jahad
          152.
          MiniOzoneHAClusterImpl frequently hangs when writing keys to bucket. Sub-task Resolved Christos Bisias
          153.
          Snapshot Diff should return path relative to bucket root Sub-task Resolved Swaminathan Balachandran
          154.
          [Snapshot] Add sequence number to all compaction log entries Sub-task Resolved Hemant Kumar
          155.
          [Snapshot] SstFilteringService should be stopped before copying incremental snapshot Sub-task Resolved Swaminathan Balachandran
          156.
          Ignore newer compaction log entries when bootstrapping followers Sub-task Resolved Hemant Kumar
          157.
          [Snapshot] SstFilteringService should use SnapshotCache Sub-task Resolved Swaminathan Balachandran
          158.
          Write a standalone tool to create a visual graph from compaction log files. Sub-task Resolved Hemant Kumar
          159.
          [Snapshot] Integration test to simulate staggered snapshot creation and deletion Sub-task Resolved Dave Teng
          160.
          [Snapshot] Add more integration tests and acceptance tests Sub-task Resolved JiaChi Wang
          161.
          [snapshot] SnapshotDiff should throw proper responses in case there are no differences to be listed Sub-task Resolved Hemant Kumar
          162.
          Stress test bootstrapping with thousands of snapshots and millions of keys. Sub-task Resolved Mladjan Gadzic
          163.
          [Snapshot] Check snapshot feature code coverage Sub-task Closed Unassigned
          164.
          snapDiff doesn't return correct results after leadership transfer Sub-task Resolved Hemant Kumar
          165.
          SSTFiltering service deleting wrong SST files from snapshot's DB dir Sub-task Resolved Hemant Kumar
          166.
          Make snapshot dir wait poll max timeout configurable Sub-task Resolved Swaminathan Balachandran
          167.
          Compaction DAG pruning service is not running after bootstrapping Sub-task Resolved Hemant Kumar
          168.
          Integration test for data integrity test for SST filtering service run Sub-task Resolved Hemant Kumar
          169.
          Add SSTFiltered flag in SnapshotInfo table. Sub-task Resolved Swaminathan Balachandran
          170.
          [Snapshot] SnapDiff to use tombstone entries in SST files Sub-task Resolved Swaminathan Balachandran
          171.
          Reduce time of compaction pause during bootstrapping Sub-task Resolved George Jahad
          172.
          Link rocksdb lib to Ozone rocksdb tools lib relative path instead of absolute path Sub-task Resolved Swaminathan Balachandran

          Activity

            People

              prashantpogde Prashant Pogde
              prashantpogde Prashant Pogde
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: