Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2802

Support for RW/RO snapshots in HDFS

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

        Attachments

        1. snapshot-testplan.pdf
          51 kB
          Jing Zhao
        2. Snapshots20121030.pdf
          289 kB
          Suresh Srinivas
        3. Snapshots20121018.pdf
          372 kB
          Tsz-wo Sze
        4. Snapshots20120429.pdf
          386 kB
          Suresh Srinivas
        5. Snapshots.pdf
          76 kB
          Suresh Srinivas
        6. snapshot-one-pager.pdf
          99 kB
          Hari Mankude
        7. snapshot-design.tex
          25 kB
          Aaron Myers
        8. snapshot-design.pdf
          189 kB
          Aaron Myers
        9. snap.patch
          122 kB
          Hari Mankude
        10. HDFSSnapshotsDesign.pdf
          73 kB
          Aaron Myers
        11. HDFS-2802-meeting-minutes-121101.txt
          5 kB
          David S. Wang
        12. HDFS-2802.branch-2.0510.patch
          1.07 MB
          Jing Zhao
        13. HDFS-2802.20121101.patch
          137 kB
          Suresh Srinivas
        14. h2802_20130509.patch
          1.07 MB
          Tsz-wo Sze
        15. h2802_20130508.patch
          1.06 MB
          Tsz-wo Sze
        16. h2802_20130430b.patch
          1.05 MB
          Tsz-wo Sze
        17. h2802_20130430.patch
          1.05 MB
          Tsz-wo Sze
        18. h2802_20130426.patch
          1.03 MB
          Tsz-wo Sze
        19. h2802_20130425.patch
          1.03 MB
          Tsz-wo Sze
        20. h2802_20130423.patch
          1015 kB
          Tsz-wo Sze
        21. h2802_20130422.patch
          1015 kB
          Tsz-wo Sze
        22. h2802_20130417.patch
          988 kB
          Tsz-wo Sze
        23. 2802.patch
          818 kB
          Suresh Srinivas
        24. 2802.patch
          819 kB
          Tsz-wo Sze
        25. 2802.diff
          867 kB
          Suresh Srinivas

          Issue Links

          1.
          Snapshot of Being Written Files Sub-task Resolved Jing Zhao
          2.
          Support snapshot of single files Sub-task Resolved Tsz-wo Sze
          3.
          Support snapshottable INodeDirectory Sub-task Resolved Tsz-wo Sze
          4.
          Handle replication in snapshots Sub-task Resolved Tsz-wo Sze
          5.
          Add SnapshotManager Sub-task Resolved Tsz-wo Sze
          6.
          Add editlog opcodes for snapshot create and delete operations Sub-task Resolved Suresh Srinivas
          7.
          Protocol changes for snapshots Sub-task Resolved Suresh Srinivas
          8.
          provide CLI support for allow and disallow snapshot on a directory Sub-task Resolved Brandon Li
          9.
          Add editlog opcodes to allow and disallow snapshots on a directory Sub-task Resolved Brandon Li
          10.
          Protocol changes for listSnapshots functionality Sub-task Resolved Brandon Li
          11.
          Add snapshot quota to limit the number of snapshots Sub-task Resolved Tsz-wo Sze
          12.
          Update file deletion logic to support snapshot files Sub-task Resolved Tsz-wo Sze
          13.
          Add snapshot related metrics Sub-task Resolved Jing Zhao
          14.
          Add snapshot information to namenode WebUI Sub-task Closed Haohui Mai
          15.
          provide CLI support for createSnapshot Sub-task Resolved Brandon Li
          16.
          Support append to original files which are snapshotted Sub-task Resolved Tsz-wo Sze
          17.
          Test replication with snapshots Sub-task Resolved Tsz-wo Sze
          18.
          Support O(1) snapshot creation Sub-task Resolved Tsz-wo Sze
          19.
          Support snapshot of subtrees Sub-task Resolved Tsz-wo Sze
          20.
          Add audit log for snapshot operations Sub-task Resolved Jing Zhao
          21.
          Change INodeDirectory.getExistingPathINodes(..) to work with snapshots Sub-task Resolved Jing Zhao
          22.
          Complete the allowSnapshot code and add a test for it Sub-task Resolved Tsz-wo Sze
          23.
          Use a persistent data structure for snapshots Sub-task Resolved Tsz-wo Sze
          24.
          Add reading/writing snapshot information to FSImage Sub-task Resolved Jing Zhao
          25.
          Add capability to namenode to get snapshot diff Sub-task Resolved Jing Zhao
          26.
          Add testcases for testing basic snapshot functionalities Sub-task Resolved Jing Zhao
          27.
          Directory diff in snapshots Sub-task Resolved Tsz-wo Sze
          28.
          Create test for all snapshot-related metrics Sub-task Resolved Jing Zhao
          29.
          Further work on changing INodeFile.blocks to private in Snapshot branch Sub-task Resolved Tsz-wo Sze
          30.
          Deletion of snapshottable dir with snapshots should fail Sub-task Resolved Jing Zhao
          31.
          Disallow write/modify operations on files and directories in a snapshot Sub-task Resolved Brandon Li
          32.
          Complete the code for disallowSnapshot Sub-task Resolved Tsz-wo Sze
          33.
          Update inode in blocksMap when deleting original/snapshot file Sub-task Resolved Jing Zhao
          34.
          FSDirectory#hasSnapshot will NPE when deleting an empty directory Sub-task Resolved Aaron Myers
          35.
          Rename should fail when the destination dir is snapshottable and has snapshots Sub-task Resolved Jing Zhao
          36.
          Add support for scheduled automatic snapshots Sub-task Resolved Suresh Srinivas
          37.
          Add support for restoring/rolling back to a snapshot Sub-task Patch Available Ajith S
          38.
          Add snapshot information to INodesInPath Sub-task Resolved Tsz-wo Sze
          39.
          Additional snapshot tests for more complicated directory structure and modifications. Sub-task Resolved Jing Zhao
          40.
          Add a snapshot parameter to INodeDirectory.getChildrenList() Sub-task Resolved Tsz-wo Sze
          41.
          Add testcases for replication handling in snapshots Sub-task Resolved Jing Zhao
          42.
          Add a snapshot id comparator Sub-task Resolved Tsz-wo Sze
          43.
          rename getter method getMutableX and getXMutable to getXAndEnsureMutable Sub-task Resolved Tsz-wo Sze
          44.
          Support renaming of snapshots Sub-task Resolved Jing Zhao
          45.
          Listing all the current snapshottable directories Sub-task Resolved Jing Zhao
          46.
          Support deleting snapshots Sub-task Resolved Jing Zhao
          47.
          Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer Sub-task Resolved Jing Zhao
          48.
          TestSnapshot fails Sub-task Resolved Jing Zhao
          49.
          Change INode to support HDFS-4103 Sub-task Resolved Tsz-wo Sze
          50.
          Support snapshots up to the snapshot limit Sub-task Resolved Tsz-wo Sze
          51.
          After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode Sub-task Resolved Jing Zhao
          52.
          When listing snapshottable directories, only return those where the user has permission to take snapshots Sub-task Resolved Jing Zhao
          53.
          INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
          54.
          Combined diff may not be identical to the diffs applied sequentially Sub-task Resolved Tsz-wo Sze
          55.
          In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like Sub-task Resolved Tsz-wo Sze
          56.
          Add support for getting snapshot diff from DistributedFileSystem Sub-task Resolved Jing Zhao
          57.
          Add unit tests for taking snapshots while file appending Sub-task Resolved Jing Zhao
          58.
          Support snapshot in OfflineImageViewer Sub-task Resolved Jing Zhao
          59.
          Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Sub-task Resolved Jing Zhao
          60.
          The snapshot copy INode.recordModification(..) returned is never used Sub-task Resolved Tsz-wo Sze
          61.
          Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class Sub-task Resolved Tsz-wo Sze
          62.
          Support file snapshot with diff lists Sub-task Resolved Tsz-wo Sze
          63.
          Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Sub-task Resolved Tsz-wo Sze
          64.
          Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot Sub-task Resolved Tsz-wo Sze
          65.
          Eliminate the file snapshot circular linked list Sub-task Resolved Tsz-wo Sze
          66.
          Fix snapshot fsimage for file diffs Sub-task Resolved Tsz-wo Sze
          67.
          Fix snapshot diff report Sub-task Resolved Jing Zhao
          68.
          Fix file/directory/snapshot deletion for file diff Sub-task Resolved Jing Zhao
          69.
          Refactor snapshot INode methods Sub-task Resolved Tsz-wo Sze
          70.
          Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot Sub-task Resolved Tsz-wo Sze
          71.
          Update quota verification for snapshot Sub-task Resolved Tsz-wo Sze
          72.
          Add CLI support for snapshot rename, diff report, and snapshottable directory listing Sub-task Resolved Jing Zhao
          73.
          Support listing snapshots under a snapshottable directory using "ls" Sub-task Resolved Jing Zhao
          74.
          Fix INodeFile replacement, TestQuota and javac error Sub-task Resolved Tsz-wo Sze
          75.
          Update SnapshotManager#snapshottables when loading fsimage Sub-task Resolved Jing Zhao
          76.
          Decide the semantic of concat with snapshots Sub-task Resolved Tsz-wo Sze
          77.
          Update rename for snapshots Sub-task Resolved Tsz-wo Sze
          78.
          Snapshot: unprotectedSetReplication may update diskspace count incorrectly Sub-task Resolved Tsz-wo Sze
          79.
          Refactor INodesInPath to a standalone class Sub-task Resolved Tsz-wo Sze
          80.
          Fix FSDirectory#delete when INode#cleanSubtree returns 0 Sub-task Resolved Jing Zhao
          81.
          Update namespace/diskspace after deleting snapshots Sub-task Resolved Jing Zhao
          82.
          Move Diff and EnumCounters to util package Sub-task Resolved Tsz-wo Sze
          83.
          Annotate snapshot tests Sub-task Resolved Arpit Agarwal
          84.
          Restrict snapshot IDs to 24-bits wide Sub-task Resolved Arpit Agarwal
          85.
          Add snapshotdiff and LsSnapshottableDir tools to hdfs script Sub-task Resolved Arpit Agarwal
          86.
          Snapshot: Update FSImage for INodeReference Sub-task Resolved Tsz-wo Sze
          87.
          Not to use INode.getParent() when generating snapshot diff report Sub-task Resolved Jing Zhao
          88.
          Update the FilesDeleted metric while deleting file/dir in the current tree Sub-task Resolved Jing Zhao
          89.
          Fix FSImageFormat#Loader NPE and synchronization issues Sub-task Resolved Jing Zhao
          90.
          Update quota usage when deleting files/dirs that were created after taking the latest snapshot Sub-task Resolved Jing Zhao
          91.
          INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
          92.
          For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted Sub-task Resolved Jing Zhao
          93.
          Add rename test in TestSnapshot Sub-task Resolved Jing Zhao
          94.
          llegalStateException renaming file under snapshottable directory Sub-task Resolved Arpit Agarwal
          95.
          Disallow users to create file/dir named ".snapshot" Sub-task Resolved Tsz-wo Sze
          96.
          Capture renamed files/directories in snapshot diff report Sub-task Closed Jing Zhao
          97.
          Renaming file in subdirectory of a snapshotted directory does not work. Sub-task Resolved Arpit Agarwal
          98.
          Fix rename across snapshottable directories Sub-task Resolved Jing Zhao
          99.
          Snapshot: Use INode id for image serialization Sub-task Resolved Tsz-wo Sze
          100.
          Fix quota computation for rename with snapshots Sub-task Resolved Jing Zhao
          101.
          Use timestamp as the default snapshot name Sub-task Resolved Tsz-wo Sze
          102.
          Fix the undo section of rename with snapshots Sub-task Resolved Jing Zhao
          103.
          Add a transient flag to file so that transient files won't be included in any snapshot Sub-task Open Tsz-wo Sze
          104.
          disallowSnapshot does not work for root Sub-task Resolved Tsz-wo Sze
          105.
          Fix FilterFileSystem and findbugs warning in Snapshot branch Sub-task Resolved Tsz-wo Sze
          106.
          Add snapshot user guide Sub-task Resolved Tsz-wo Sze
          107.
          Change the parameter type of the snapshot methods in HdfsAdmin to Path Sub-task Resolved Tsz-wo Sze
          108.
          Minor simplifications to snapshot code Sub-task Resolved Arpit Agarwal
          109.
          Fix test failures after merging the mapping from INodeId to INode Sub-task Resolved Jing Zhao
          110.
          Update inodeMap after deleting files/directories/snapshots Sub-task Resolved Jing Zhao
          111.
          Snapshot tests broken after merge from trunk Sub-task Resolved Arpit Agarwal
          112.
          Fix OfflineImageViewer and permission checking for snapshot operations Sub-task Resolved Jing Zhao
          113.
          DisallowSnapshot throws IllegalStateException Sub-task Resolved Jing Zhao
          114.
          Fix javadoc and other warnings in the snapshot branch Sub-task Resolved Tsz-wo Sze
          115.
          Fix appending to a renamed file with snapshot Sub-task Resolved Jing Zhao
          116.
          Convert snapshot user guide to APT from XDOC Sub-task Resolved Aaron Myers
          117.
          Use INodeId to identify the corresponding directory node for FSImage saving/loading Sub-task Resolved Jing Zhao
          118.
          AccessControlException message is changed in snapshot branch Sub-task Resolved Tsz-wo Sze
          119.
          Disallow nested snapshottable directories Sub-task Resolved Tsz-wo Sze
          120.
          snapshotDiff of two invalid snapshots but with same name returns success Sub-task Resolved Jing Zhao
          121.
          Update inodeMap after node replacement Sub-task Resolved Jing Zhao
          122.
          directory is not snapshottable after clrQuota Sub-task Resolved Jing Zhao
          123.
          Fix bugs in quota usage updating/computation Sub-task Resolved Jing Zhao
          124.
          File listing of .snapshot under a non-existing dir throws NullPointer Sub-task Resolved Tsz-wo Sze
          125.
          Update and fix deletion of reference inode Sub-task Resolved Jing Zhao
          126.
          Update computeContentSummary() for the reference nodes in snapshots Sub-task Resolved Tsz-wo Sze
          127.
          Fix INodeDirectoryWithSnapshot#cleanDeletedINode Sub-task Resolved Jing Zhao
          128.
          lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable Sub-task Resolved Jing Zhao
          129.
          Disallowing snapshot on / twice throws error Sub-task Resolved Jing Zhao
          130.
          For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff Sub-task Resolved Jing Zhao
          131.
          Add more unit tests for the undo section of rename Sub-task Resolved Jing Zhao
          132.
          Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree Sub-task Closed Jing Zhao
          133.
          The javadoc of FileWithSnapshot is incorrect Sub-task Closed Tsz-wo Sze
          134.
          Add a test for testing snapshot file length Sub-task Closed Arpit Agarwal
          135.
          Update Snapshot doc for HDFS-4758 Sub-task Closed Tsz-wo Sze
          136.
          Reduce snapshot inode memory usage Sub-task Closed Tsz-wo Sze

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                harip Hari Mankude
              • Votes:
                1 Vote for this issue
                Watchers:
                79 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: