Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

      1. snapshot-testplan.pdf
        51 kB
        Jing Zhao
      2. Snapshots20121030.pdf
        289 kB
        Suresh Srinivas
      3. Snapshots20121018.pdf
        372 kB
        Tsz Wo Nicholas Sze
      4. Snapshots20120429.pdf
        386 kB
        Suresh Srinivas
      5. Snapshots.pdf
        76 kB
        Suresh Srinivas
      6. snapshot-one-pager.pdf
        99 kB
        Hari Mankude
      7. snapshot-design.tex
        25 kB
        Aaron T. Myers
      8. snapshot-design.pdf
        189 kB
        Aaron T. Myers
      9. snap.patch
        122 kB
        Hari Mankude
      10. HDFSSnapshotsDesign.pdf
        73 kB
        Aaron T. Myers
      11. HDFS-2802-meeting-minutes-121101.txt
        5 kB
        David S. Wang
      12. HDFS-2802.branch-2.0510.patch
        1.07 MB
        Jing Zhao
      13. HDFS-2802.20121101.patch
        137 kB
        Suresh Srinivas
      14. h2802_20130509.patch
        1.07 MB
        Tsz Wo Nicholas Sze
      15. h2802_20130508.patch
        1.06 MB
        Tsz Wo Nicholas Sze
      16. h2802_20130430b.patch
        1.05 MB
        Tsz Wo Nicholas Sze
      17. h2802_20130430.patch
        1.05 MB
        Tsz Wo Nicholas Sze
      18. h2802_20130426.patch
        1.03 MB
        Tsz Wo Nicholas Sze
      19. h2802_20130425.patch
        1.03 MB
        Tsz Wo Nicholas Sze
      20. h2802_20130423.patch
        1015 kB
        Tsz Wo Nicholas Sze
      21. h2802_20130422.patch
        1015 kB
        Tsz Wo Nicholas Sze
      22. h2802_20130417.patch
        988 kB
        Tsz Wo Nicholas Sze
      23. 2802.patch
        818 kB
        Suresh Srinivas
      24. 2802.patch
        819 kB
        Tsz Wo Nicholas Sze
      25. 2802.diff
        867 kB
        Suresh Srinivas

        Issue Links

        1.
        Snapshot of Being Written Files Sub-task Resolved Jing Zhao
         
        2.
        Support snapshot of single files Sub-task Resolved Tsz Wo Nicholas Sze
         
        3.
        Support snapshottable INodeDirectory Sub-task Resolved Tsz Wo Nicholas Sze
         
        4.
        Handle replication in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        5.
        Add SnapshotManager Sub-task Resolved Tsz Wo Nicholas Sze
         
        6.
        Add editlog opcodes for snapshot create and delete operations Sub-task Resolved Suresh Srinivas
         
        7.
        Protocol changes for snapshots Sub-task Resolved Suresh Srinivas
         
        8.
        provide CLI support for allow and disallow snapshot on a directory Sub-task Resolved Brandon Li
         
        9.
        Add editlog opcodes to allow and disallow snapshots on a directory Sub-task Resolved Brandon Li
         
        10.
        Protocol changes for listSnapshots functionality Sub-task Resolved Brandon Li
         
        11.
        Add snapshot quota to limit the number of snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        12.
        Update file deletion logic to support snapshot files Sub-task Resolved Tsz Wo Nicholas Sze
         
        13.
        Add snapshot related metrics Sub-task Resolved Jing Zhao
         
        14.
        Add snapshot information to namenode WebUI Sub-task Closed Haohui Mai
         
        15.
        provide CLI support for createSnapshot Sub-task Resolved Brandon Li
         
        16.
        Support append to original files which are snapshotted Sub-task Resolved Tsz Wo Nicholas Sze
         
        17.
        Test replication with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        18.
        Support O(1) snapshot creation Sub-task Resolved Tsz Wo Nicholas Sze
         
        19.
        Support snapshot of subtrees Sub-task Resolved Tsz Wo Nicholas Sze
         
        20.
        Add audit log for snapshot operations Sub-task Resolved Jing Zhao
         
        21.
        Change INodeDirectory.getExistingPathINodes(..) to work with snapshots Sub-task Resolved Jing Zhao
         
        22.
        Complete the allowSnapshot code and add a test for it Sub-task Resolved Tsz Wo Nicholas Sze
         
        23.
        Use a persistent data structure for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        24.
        Add reading/writing snapshot information to FSImage Sub-task Resolved Jing Zhao
         
        25.
        Add capability to namenode to get snapshot diff Sub-task Resolved Jing Zhao
         
        26.
        Add testcases for testing basic snapshot functionalities Sub-task Resolved Jing Zhao
         
        27.
        Directory diff in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        28.
        Create test for all snapshot-related metrics Sub-task Resolved Jing Zhao
         
        29.
        Further work on changing INodeFile.blocks to private in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        30.
        Deletion of snapshottable dir with snapshots should fail Sub-task Resolved Jing Zhao
         
        31.
        Disallow write/modify operations on files and directories in a snapshot Sub-task Resolved Brandon Li
         
        32.
        Complete the code for disallowSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        33.
        Update inode in blocksMap when deleting original/snapshot file Sub-task Resolved Jing Zhao
         
        34.
        FSDirectory#hasSnapshot will NPE when deleting an empty directory Sub-task Resolved Aaron T. Myers
         
        35.
        Rename should fail when the destination dir is snapshottable and has snapshots Sub-task Resolved Jing Zhao
         
        36.
        Add support for scheduled automatic snapshots Sub-task Resolved Suresh Srinivas
         
        37. Add support for restoring/rolling back to a snapshot Sub-task Open Jing Zhao
         
        38.
        Add snapshot information to INodesInPath Sub-task Resolved Tsz Wo Nicholas Sze
         
        39.
        Additional snapshot tests for more complicated directory structure and modifications. Sub-task Resolved Jing Zhao
         
        40.
        Add a snapshot parameter to INodeDirectory.getChildrenList() Sub-task Resolved Tsz Wo Nicholas Sze
         
        41.
        Add testcases for replication handling in snapshots Sub-task Resolved Jing Zhao
         
        42.
        Add a snapshot id comparator Sub-task Resolved Tsz Wo Nicholas Sze
         
        43.
        rename getter method getMutableX and getXMutable to getXAndEnsureMutable Sub-task Resolved Tsz Wo Nicholas Sze
         
        44.
        Support renaming of snapshots Sub-task Resolved Jing Zhao
         
        45.
        Listing all the current snapshottable directories Sub-task Resolved Jing Zhao
         
        46.
        Support deleting snapshots Sub-task Resolved Jing Zhao
         
        47.
        Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer Sub-task Resolved Jing Zhao
         
        48.
        TestSnapshot fails Sub-task Resolved Jing Zhao
         
        49.
        Change INode to support HDFS-4103 Sub-task Resolved Tsz Wo Nicholas Sze
         
        50.
        Support snapshots up to the snapshot limit Sub-task Resolved Tsz Wo Nicholas Sze
         
        51.
        After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode Sub-task Resolved Jing Zhao
         
        52.
        When listing snapshottable directories, only return those where the user has permission to take snapshots Sub-task Resolved Jing Zhao
         
        53.
        INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
         
        54.
        Combined diff may not be identical to the diffs applied sequentially Sub-task Resolved Tsz Wo Nicholas Sze
         
        55.
        In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like Sub-task Resolved Tsz Wo Nicholas Sze
         
        56.
        Add support for getting snapshot diff from DistributedFileSystem Sub-task Resolved Jing Zhao
         
        57.
        Add unit tests for taking snapshots while file appending Sub-task Resolved Jing Zhao
         
        58.
        Support snapshot in OfflineImageViewer Sub-task Resolved Jing Zhao
         
        59.
        Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Sub-task Resolved Jing Zhao
         
        60.
        The snapshot copy INode.recordModification(..) returned is never used Sub-task Resolved Tsz Wo Nicholas Sze
         
        61.
        Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class Sub-task Resolved Tsz Wo Nicholas Sze
         
        62.
        Support file snapshot with diff lists Sub-task Resolved Tsz Wo Nicholas Sze
         
        63.
        Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        64.
        Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        65.
        Eliminate the file snapshot circular linked list Sub-task Resolved Tsz Wo Nicholas Sze
         
        66.
        Fix snapshot fsimage for file diffs Sub-task Resolved Tsz Wo Nicholas Sze
         
        67.
        Fix snapshot diff report Sub-task Resolved Jing Zhao
         
        68.
        Fix file/directory/snapshot deletion for file diff Sub-task Resolved Jing Zhao
         
        69.
        Refactor snapshot INode methods Sub-task Resolved Tsz Wo Nicholas Sze
         
        70.
        Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        71.
        Update quota verification for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        72.
        Add CLI support for snapshot rename, diff report, and snapshottable directory listing Sub-task Resolved Jing Zhao
         
        73.
        Support listing snapshots under a snapshottable directory using "ls" Sub-task Resolved Jing Zhao
         
        74.
        Fix INodeFile replacement, TestQuota and javac error Sub-task Resolved Tsz Wo Nicholas Sze
         
        75.
        Update SnapshotManager#snapshottables when loading fsimage Sub-task Resolved Jing Zhao
         
        76.
        Decide the semantic of concat with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        77.
        Update rename for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        78.
        Snapshot: unprotectedSetReplication may update diskspace count incorrectly Sub-task Resolved Tsz Wo Nicholas Sze
         
        79.
        Refactor INodesInPath to a standalone class Sub-task Resolved Tsz Wo Nicholas Sze
         
        80.
        Fix FSDirectory#delete when INode#cleanSubtree returns 0 Sub-task Resolved Jing Zhao
         
        81.
        Update namespace/diskspace after deleting snapshots Sub-task Resolved Jing Zhao
         
        82.
        Move Diff and EnumCounters to util package Sub-task Resolved Tsz Wo Nicholas Sze
         
        83.
        Annotate snapshot tests Sub-task Resolved Arpit Agarwal
         
        84.
        Restrict snapshot IDs to 24-bits wide Sub-task Resolved Arpit Agarwal
         
        85.
        Add snapshotdiff and LsSnapshottableDir tools to hdfs script Sub-task Resolved Arpit Agarwal
         
        86.
        Snapshot: Update FSImage for INodeReference Sub-task Resolved Tsz Wo Nicholas Sze
         
        87.
        Not to use INode.getParent() when generating snapshot diff report Sub-task Resolved Jing Zhao
         
        88.
        Update the FilesDeleted metric while deleting file/dir in the current tree Sub-task Resolved Jing Zhao
         
        89.
        Fix FSImageFormat#Loader NPE and synchronization issues Sub-task Resolved Jing Zhao
         
        90.
        Update quota usage when deleting files/dirs that were created after taking the latest snapshot Sub-task Resolved Jing Zhao
         
        91.
        INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
         
        92.
        For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted Sub-task Resolved Jing Zhao
         
        93.
        Add rename test in TestSnapshot Sub-task Resolved Jing Zhao
         
        94.
        llegalStateException renaming file under snapshottable directory Sub-task Resolved Arpit Agarwal
         
        95.
        Disallow users to create file/dir named ".snapshot" Sub-task Resolved Tsz Wo Nicholas Sze
         
        96.
        Capture renamed files/directories in snapshot diff report Sub-task Closed Jing Zhao
         
        97.
        Renaming file in subdirectory of a snapshotted directory does not work. Sub-task Resolved Arpit Agarwal
         
        98.
        Fix rename across snapshottable directories Sub-task Resolved Jing Zhao
         
        99.
        Snapshot: Use INode id for image serialization Sub-task Resolved Tsz Wo Nicholas Sze
         
        100.
        Fix quota computation for rename with snapshots Sub-task Resolved Jing Zhao
         
        101.
        Use timestamp as the default snapshot name Sub-task Resolved Tsz Wo Nicholas Sze
         
        102.
        Fix the undo section of rename with snapshots Sub-task Resolved Jing Zhao
         
        103. Add a transient flag to file so that transient files won't be included in any snapshot Sub-task Open Tsz Wo Nicholas Sze
         
        104.
        disallowSnapshot does not work for root Sub-task Resolved Tsz Wo Nicholas Sze
         
        105.
        Fix FilterFileSystem and findbugs warning in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        106.
        Add snapshot user guide Sub-task Resolved Tsz Wo Nicholas Sze
         
        107.
        Change the parameter type of the snapshot methods in HdfsAdmin to Path Sub-task Resolved Tsz Wo Nicholas Sze
         
        108.
        Minor simplifications to snapshot code Sub-task Resolved Arpit Agarwal
         
        109.
        Fix test failures after merging the mapping from INodeId to INode Sub-task Resolved Jing Zhao
         
        110.
        Update inodeMap after deleting files/directories/snapshots Sub-task Resolved Jing Zhao
         
        111.
        Snapshot tests broken after merge from trunk Sub-task Resolved Arpit Agarwal
         
        112.
        Fix OfflineImageViewer and permission checking for snapshot operations Sub-task Resolved Jing Zhao
         
        113.
        DisallowSnapshot throws IllegalStateException Sub-task Resolved Jing Zhao
         
        114.
        Fix javadoc and other warnings in the snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        115.
        Fix appending to a renamed file with snapshot Sub-task Resolved Jing Zhao
         
        116.
        Convert snapshot user guide to APT from XDOC Sub-task Resolved Aaron T. Myers
         
        117.
        Use INodeId to identify the corresponding directory node for FSImage saving/loading Sub-task Resolved Jing Zhao
         
        118.
        AccessControlException message is changed in snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        119.
        Disallow nested snapshottable directories Sub-task Resolved Tsz Wo Nicholas Sze
         
        120.
        snapshotDiff of two invalid snapshots but with same name returns success Sub-task Resolved Jing Zhao
         
        121.
        Update inodeMap after node replacement Sub-task Resolved Jing Zhao
         
        122.
        directory is not snapshottable after clrQuota Sub-task Resolved Jing Zhao
         
        123.
        Fix bugs in quota usage updating/computation Sub-task Resolved Jing Zhao
         
        124.
        File listing of .snapshot under a non-existing dir throws NullPointer Sub-task Resolved Tsz Wo Nicholas Sze
         
        125.
        Update and fix deletion of reference inode Sub-task Resolved Jing Zhao
         
        126.
        Update computeContentSummary() for the reference nodes in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        127.
        Fix INodeDirectoryWithSnapshot#cleanDeletedINode Sub-task Resolved Jing Zhao
         
        128.
        lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable Sub-task Resolved Jing Zhao
         
        129.
        Disallowing snapshot on / twice throws error Sub-task Resolved Jing Zhao
         
        130.
        For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff Sub-task Resolved Jing Zhao
         
        131.
        Add more unit tests for the undo section of rename Sub-task Resolved Jing Zhao
         
        132.
        Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree Sub-task Closed Jing Zhao
         
        133.
        The javadoc of FileWithSnapshot is incorrect Sub-task Closed Tsz Wo Nicholas Sze
         
        134.
        Add a test for testing snapshot file length Sub-task Closed Arpit Agarwal
         
        135.
        Update Snapshot doc for HDFS-4758 Sub-task Closed Tsz Wo Nicholas Sze
         
        136.
        Reduce snapshot inode memory usage Sub-task Closed Tsz Wo Nicholas Sze
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Hari Mankude
            • Votes:
              1 Vote for this issue
              Watchers:
              72 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development