Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

        Attachments

        1. snapshot-testplan.pdf
          51 kB
          Jing Zhao
        2. Snapshots20121030.pdf
          289 kB
          Suresh Srinivas
        3. Snapshots20121018.pdf
          372 kB
          Tsz Wo Nicholas Sze
        4. Snapshots20120429.pdf
          386 kB
          Suresh Srinivas
        5. Snapshots.pdf
          76 kB
          Suresh Srinivas
        6. snapshot-one-pager.pdf
          99 kB
          Hari Mankude
        7. snapshot-design.tex
          25 kB
          Aaron T. Myers
        8. snapshot-design.pdf
          189 kB
          Aaron T. Myers
        9. snap.patch
          122 kB
          Hari Mankude
        10. HDFSSnapshotsDesign.pdf
          73 kB
          Aaron T. Myers
        11. HDFS-2802-meeting-minutes-121101.txt
          5 kB
          David S. Wang
        12. HDFS-2802.branch-2.0510.patch
          1.07 MB
          Jing Zhao
        13. HDFS-2802.20121101.patch
          137 kB
          Suresh Srinivas
        14. h2802_20130509.patch
          1.07 MB
          Tsz Wo Nicholas Sze
        15. h2802_20130508.patch
          1.06 MB
          Tsz Wo Nicholas Sze
        16. h2802_20130430b.patch
          1.05 MB
          Tsz Wo Nicholas Sze
        17. h2802_20130430.patch
          1.05 MB
          Tsz Wo Nicholas Sze
        18. h2802_20130426.patch
          1.03 MB
          Tsz Wo Nicholas Sze
        19. h2802_20130425.patch
          1.03 MB
          Tsz Wo Nicholas Sze
        20. h2802_20130423.patch
          1015 kB
          Tsz Wo Nicholas Sze
        21. h2802_20130422.patch
          1015 kB
          Tsz Wo Nicholas Sze
        22. h2802_20130417.patch
          988 kB
          Tsz Wo Nicholas Sze
        23. 2802.patch
          818 kB
          Suresh Srinivas
        24. 2802.patch
          819 kB
          Tsz Wo Nicholas Sze
        25. 2802.diff
          867 kB
          Suresh Srinivas

          Issue Links

          1.
          Snapshot of Being Written Files Sub-task Resolved Jing Zhao
          2.
          Support snapshot of single files Sub-task Resolved Tsz Wo Nicholas Sze
          3.
          Support snapshottable INodeDirectory Sub-task Resolved Tsz Wo Nicholas Sze
          4.
          Handle replication in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          5.
          Add SnapshotManager Sub-task Resolved Tsz Wo Nicholas Sze
          6.
          Add editlog opcodes for snapshot create and delete operations Sub-task Resolved Suresh Srinivas
          7.
          Protocol changes for snapshots Sub-task Resolved Suresh Srinivas
          8.
          provide CLI support for allow and disallow snapshot on a directory Sub-task Resolved Brandon Li
          9.
          Add editlog opcodes to allow and disallow snapshots on a directory Sub-task Resolved Brandon Li
          10.
          Protocol changes for listSnapshots functionality Sub-task Resolved Brandon Li
          11.
          Add snapshot quota to limit the number of snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          12.
          Update file deletion logic to support snapshot files Sub-task Resolved Tsz Wo Nicholas Sze
          13.
          Add snapshot related metrics Sub-task Resolved Jing Zhao
          14.
          Add snapshot information to namenode WebUI Sub-task Closed Haohui Mai
          15.
          provide CLI support for createSnapshot Sub-task Resolved Brandon Li
          16.
          Support append to original files which are snapshotted Sub-task Resolved Tsz Wo Nicholas Sze
          17.
          Test replication with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          18.
          Support O(1) snapshot creation Sub-task Resolved Tsz Wo Nicholas Sze
          19.
          Support snapshot of subtrees Sub-task Resolved Tsz Wo Nicholas Sze
          20.
          Add audit log for snapshot operations Sub-task Resolved Jing Zhao
          21.
          Change INodeDirectory.getExistingPathINodes(..) to work with snapshots Sub-task Resolved Jing Zhao
          22.
          Complete the allowSnapshot code and add a test for it Sub-task Resolved Tsz Wo Nicholas Sze
          23.
          Use a persistent data structure for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          24.
          Add reading/writing snapshot information to FSImage Sub-task Resolved Jing Zhao
          25.
          Add capability to namenode to get snapshot diff Sub-task Resolved Jing Zhao
          26.
          Add testcases for testing basic snapshot functionalities Sub-task Resolved Jing Zhao
          27.
          Directory diff in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          28.
          Create test for all snapshot-related metrics Sub-task Resolved Jing Zhao
          29.
          Further work on changing INodeFile.blocks to private in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
          30.
          Deletion of snapshottable dir with snapshots should fail Sub-task Resolved Jing Zhao
          31.
          Disallow write/modify operations on files and directories in a snapshot Sub-task Resolved Brandon Li
          32.
          Complete the code for disallowSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
          33.
          Update inode in blocksMap when deleting original/snapshot file Sub-task Resolved Jing Zhao
          34.
          FSDirectory#hasSnapshot will NPE when deleting an empty directory Sub-task Resolved Aaron T. Myers
          35.
          Rename should fail when the destination dir is snapshottable and has snapshots Sub-task Resolved Jing Zhao
          36.
          Add support for scheduled automatic snapshots Sub-task Resolved Suresh Srinivas
          37.
          Add support for restoring/rolling back to a snapshot Sub-task Patch Available Ajith S
          38.
          Add snapshot information to INodesInPath Sub-task Resolved Tsz Wo Nicholas Sze
          39.
          Additional snapshot tests for more complicated directory structure and modifications. Sub-task Resolved Jing Zhao
          40.
          Add a snapshot parameter to INodeDirectory.getChildrenList() Sub-task Resolved Tsz Wo Nicholas Sze
          41.
          Add testcases for replication handling in snapshots Sub-task Resolved Jing Zhao
          42.
          Add a snapshot id comparator Sub-task Resolved Tsz Wo Nicholas Sze
          43.
          rename getter method getMutableX and getXMutable to getXAndEnsureMutable Sub-task Resolved Tsz Wo Nicholas Sze
          44.
          Support renaming of snapshots Sub-task Resolved Jing Zhao
          45.
          Listing all the current snapshottable directories Sub-task Resolved Jing Zhao
          46.
          Support deleting snapshots Sub-task Resolved Jing Zhao
          47.
          Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer Sub-task Resolved Jing Zhao
          48.
          TestSnapshot fails Sub-task Resolved Jing Zhao
          49.
          Change INode to support HDFS-4103 Sub-task Resolved Tsz Wo Nicholas Sze
          50.
          Support snapshots up to the snapshot limit Sub-task Resolved Tsz Wo Nicholas Sze
          51.
          After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode Sub-task Resolved Jing Zhao
          52.
          When listing snapshottable directories, only return those where the user has permission to take snapshots Sub-task Resolved Jing Zhao
          53.
          INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
          54.
          Combined diff may not be identical to the diffs applied sequentially Sub-task Resolved Tsz Wo Nicholas Sze
          55.
          In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like Sub-task Resolved Tsz Wo Nicholas Sze
          56.
          Add support for getting snapshot diff from DistributedFileSystem Sub-task Resolved Jing Zhao
          57.
          Add unit tests for taking snapshots while file appending Sub-task Resolved Jing Zhao
          58.
          Support snapshot in OfflineImageViewer Sub-task Resolved Jing Zhao
          59.
          Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Sub-task Resolved Jing Zhao
          60.
          The snapshot copy INode.recordModification(..) returned is never used Sub-task Resolved Tsz Wo Nicholas Sze
          61.
          Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class Sub-task Resolved Tsz Wo Nicholas Sze
          62.
          Support file snapshot with diff lists Sub-task Resolved Tsz Wo Nicholas Sze
          63.
          Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
          64.
          Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
          65.
          Eliminate the file snapshot circular linked list Sub-task Resolved Tsz Wo Nicholas Sze
          66.
          Fix snapshot fsimage for file diffs Sub-task Resolved Tsz Wo Nicholas Sze
          67.
          Fix snapshot diff report Sub-task Resolved Jing Zhao
          68.
          Fix file/directory/snapshot deletion for file diff Sub-task Resolved Jing Zhao
          69.
          Refactor snapshot INode methods Sub-task Resolved Tsz Wo Nicholas Sze
          70.
          Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
          71.
          Update quota verification for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
          72.
          Add CLI support for snapshot rename, diff report, and snapshottable directory listing Sub-task Resolved Jing Zhao
          73.
          Support listing snapshots under a snapshottable directory using "ls" Sub-task Resolved Jing Zhao
          74.
          Fix INodeFile replacement, TestQuota and javac error Sub-task Resolved Tsz Wo Nicholas Sze
          75.
          Update SnapshotManager#snapshottables when loading fsimage Sub-task Resolved Jing Zhao
          76.
          Decide the semantic of concat with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          77.
          Update rename for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          78.
          Snapshot: unprotectedSetReplication may update diskspace count incorrectly Sub-task Resolved Tsz Wo Nicholas Sze
          79.
          Refactor INodesInPath to a standalone class Sub-task Resolved Tsz Wo Nicholas Sze
          80.
          Fix FSDirectory#delete when INode#cleanSubtree returns 0 Sub-task Resolved Jing Zhao
          81.
          Update namespace/diskspace after deleting snapshots Sub-task Resolved Jing Zhao
          82.
          Move Diff and EnumCounters to util package Sub-task Resolved Tsz Wo Nicholas Sze
          83.
          Annotate snapshot tests Sub-task Resolved Arpit Agarwal
          84.
          Restrict snapshot IDs to 24-bits wide Sub-task Resolved Arpit Agarwal
          85.
          Add snapshotdiff and LsSnapshottableDir tools to hdfs script Sub-task Resolved Arpit Agarwal
          86.
          Snapshot: Update FSImage for INodeReference Sub-task Resolved Tsz Wo Nicholas Sze
          87.
          Not to use INode.getParent() when generating snapshot diff report Sub-task Resolved Jing Zhao
          88.
          Update the FilesDeleted metric while deleting file/dir in the current tree Sub-task Resolved Jing Zhao
          89.
          Fix FSImageFormat#Loader NPE and synchronization issues Sub-task Resolved Jing Zhao
          90.
          Update quota usage when deleting files/dirs that were created after taking the latest snapshot Sub-task Resolved Jing Zhao
          91.
          INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
          92.
          For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted Sub-task Resolved Jing Zhao
          93.
          Add rename test in TestSnapshot Sub-task Resolved Jing Zhao
          94.
          llegalStateException renaming file under snapshottable directory Sub-task Resolved Arpit Agarwal
          95.
          Disallow users to create file/dir named ".snapshot" Sub-task Resolved Tsz Wo Nicholas Sze
          96.
          Capture renamed files/directories in snapshot diff report Sub-task Closed Jing Zhao
          97.
          Renaming file in subdirectory of a snapshotted directory does not work. Sub-task Resolved Arpit Agarwal
          98.
          Fix rename across snapshottable directories Sub-task Resolved Jing Zhao
          99.
          Snapshot: Use INode id for image serialization Sub-task Resolved Tsz Wo Nicholas Sze
          100.
          Fix quota computation for rename with snapshots Sub-task Resolved Jing Zhao
          101.
          Use timestamp as the default snapshot name Sub-task Resolved Tsz Wo Nicholas Sze
          102.
          Fix the undo section of rename with snapshots Sub-task Resolved Jing Zhao
          103.
          Add a transient flag to file so that transient files won't be included in any snapshot Sub-task Open Tsz Wo Nicholas Sze
          104.
          disallowSnapshot does not work for root Sub-task Resolved Tsz Wo Nicholas Sze
          105.
          Fix FilterFileSystem and findbugs warning in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
          106.
          Add snapshot user guide Sub-task Resolved Tsz Wo Nicholas Sze
          107.
          Change the parameter type of the snapshot methods in HdfsAdmin to Path Sub-task Resolved Tsz Wo Nicholas Sze
          108.
          Minor simplifications to snapshot code Sub-task Resolved Arpit Agarwal
          109.
          Fix test failures after merging the mapping from INodeId to INode Sub-task Resolved Jing Zhao
          110.
          Update inodeMap after deleting files/directories/snapshots Sub-task Resolved Jing Zhao
          111.
          Snapshot tests broken after merge from trunk Sub-task Resolved Arpit Agarwal
          112.
          Fix OfflineImageViewer and permission checking for snapshot operations Sub-task Resolved Jing Zhao
          113.
          DisallowSnapshot throws IllegalStateException Sub-task Resolved Jing Zhao
          114.
          Fix javadoc and other warnings in the snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
          115.
          Fix appending to a renamed file with snapshot Sub-task Resolved Jing Zhao
          116.
          Convert snapshot user guide to APT from XDOC Sub-task Resolved Aaron T. Myers
          117.
          Use INodeId to identify the corresponding directory node for FSImage saving/loading Sub-task Resolved Jing Zhao
          118.
          AccessControlException message is changed in snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
          119.
          Disallow nested snapshottable directories Sub-task Resolved Tsz Wo Nicholas Sze
          120.
          snapshotDiff of two invalid snapshots but with same name returns success Sub-task Resolved Jing Zhao
          121.
          Update inodeMap after node replacement Sub-task Resolved Jing Zhao
          122.
          directory is not snapshottable after clrQuota Sub-task Resolved Jing Zhao
          123.
          Fix bugs in quota usage updating/computation Sub-task Resolved Jing Zhao
          124.
          File listing of .snapshot under a non-existing dir throws NullPointer Sub-task Resolved Tsz Wo Nicholas Sze
          125.
          Update and fix deletion of reference inode Sub-task Resolved Jing Zhao
          126.
          Update computeContentSummary() for the reference nodes in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
          127.
          Fix INodeDirectoryWithSnapshot#cleanDeletedINode Sub-task Resolved Jing Zhao
          128.
          lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable Sub-task Resolved Jing Zhao
          129.
          Disallowing snapshot on / twice throws error Sub-task Resolved Jing Zhao
          130.
          For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff Sub-task Resolved Jing Zhao
          131.
          Add more unit tests for the undo section of rename Sub-task Resolved Jing Zhao
          132.
          Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree Sub-task Closed Jing Zhao
          133.
          The javadoc of FileWithSnapshot is incorrect Sub-task Closed Tsz Wo Nicholas Sze
          134.
          Add a test for testing snapshot file length Sub-task Closed Arpit Agarwal
          135.
          Update Snapshot doc for HDFS-4758 Sub-task Closed Tsz Wo Nicholas Sze
          136.
          Reduce snapshot inode memory usage Sub-task Closed Tsz Wo Nicholas Sze

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                harip Hari Mankude
              • Votes:
                1 Vote for this issue
                Watchers:
                78 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: