Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2802

Support for RW/RO snapshots in HDFS

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

        Attachments

        1. 2802.diff
          867 kB
          Suresh Srinivas
        2. 2802.patch
          819 kB
          Tsz-wo Sze
        3. 2802.patch
          818 kB
          Suresh Srinivas
        4. h2802_20130417.patch
          988 kB
          Tsz-wo Sze
        5. h2802_20130422.patch
          1015 kB
          Tsz-wo Sze
        6. h2802_20130423.patch
          1015 kB
          Tsz-wo Sze
        7. h2802_20130425.patch
          1.03 MB
          Tsz-wo Sze
        8. h2802_20130426.patch
          1.03 MB
          Tsz-wo Sze
        9. h2802_20130430.patch
          1.05 MB
          Tsz-wo Sze
        10. h2802_20130430b.patch
          1.05 MB
          Tsz-wo Sze
        11. h2802_20130508.patch
          1.06 MB
          Tsz-wo Sze
        12. h2802_20130509.patch
          1.07 MB
          Tsz-wo Sze
        13. HDFS-2802.20121101.patch
          137 kB
          Suresh Srinivas
        14. HDFS-2802.branch-2.0510.patch
          1.07 MB
          Jing Zhao
        15. HDFS-2802-meeting-minutes-121101.txt
          5 kB
          David S. Wang
        16. HDFSSnapshotsDesign.pdf
          73 kB
          Aaron Myers
        17. snap.patch
          122 kB
          Hari Mankude
        18. snapshot-design.pdf
          189 kB
          Aaron Myers
        19. snapshot-design.tex
          25 kB
          Aaron Myers
        20. snapshot-one-pager.pdf
          99 kB
          Hari Mankude
        21. Snapshots.pdf
          76 kB
          Suresh Srinivas
        22. Snapshots20120429.pdf
          386 kB
          Suresh Srinivas
        23. Snapshots20121018.pdf
          372 kB
          Tsz-wo Sze
        24. Snapshots20121030.pdf
          289 kB
          Suresh Srinivas
        25. snapshot-testplan.pdf
          51 kB
          Jing Zhao

        Issue Links

        1.
        Snapshot of Being Written Files Sub-task Resolved Jing Zhao Actions
        2.
        Support snapshot of single files Sub-task Resolved Tsz-wo Sze Actions
        3.
        Support snapshottable INodeDirectory Sub-task Resolved Tsz-wo Sze Actions
        4.
        Handle replication in snapshots Sub-task Resolved Tsz-wo Sze Actions
        5.
        Add SnapshotManager Sub-task Resolved Tsz-wo Sze Actions
        6.
        Add editlog opcodes for snapshot create and delete operations Sub-task Resolved Suresh Srinivas Actions
        7.
        Protocol changes for snapshots Sub-task Resolved Suresh Srinivas Actions
        8.
        provide CLI support for allow and disallow snapshot on a directory Sub-task Resolved Brandon Li Actions
        9.
        Add editlog opcodes to allow and disallow snapshots on a directory Sub-task Resolved Brandon Li Actions
        10.
        Protocol changes for listSnapshots functionality Sub-task Resolved Brandon Li Actions
        11.
        Add snapshot quota to limit the number of snapshots Sub-task Resolved Tsz-wo Sze Actions
        12.
        Update file deletion logic to support snapshot files Sub-task Resolved Tsz-wo Sze Actions
        13.
        Add snapshot related metrics Sub-task Resolved Jing Zhao Actions
        14.
        Add snapshot information to namenode WebUI Sub-task Closed Haohui Mai Actions
        15.
        provide CLI support for createSnapshot Sub-task Resolved Brandon Li Actions
        16.
        Support append to original files which are snapshotted Sub-task Resolved Tsz-wo Sze Actions
        17.
        Test replication with snapshots Sub-task Resolved Tsz-wo Sze Actions
        18.
        Support O(1) snapshot creation Sub-task Resolved Tsz-wo Sze Actions
        19.
        Support snapshot of subtrees Sub-task Resolved Tsz-wo Sze Actions
        20.
        Add audit log for snapshot operations Sub-task Resolved Jing Zhao Actions
        21.
        Change INodeDirectory.getExistingPathINodes(..) to work with snapshots Sub-task Resolved Jing Zhao Actions
        22.
        Complete the allowSnapshot code and add a test for it Sub-task Resolved Tsz-wo Sze Actions
        23.
        Use a persistent data structure for snapshots Sub-task Resolved Tsz-wo Sze Actions
        24.
        Add reading/writing snapshot information to FSImage Sub-task Resolved Jing Zhao Actions
        25.
        Add capability to namenode to get snapshot diff Sub-task Resolved Jing Zhao Actions
        26.
        Add testcases for testing basic snapshot functionalities Sub-task Resolved Jing Zhao Actions
        27.
        Directory diff in snapshots Sub-task Resolved Tsz-wo Sze Actions
        28.
        Create test for all snapshot-related metrics Sub-task Resolved Jing Zhao Actions
        29.
        Further work on changing INodeFile.blocks to private in Snapshot branch Sub-task Resolved Tsz-wo Sze Actions
        30.
        Deletion of snapshottable dir with snapshots should fail Sub-task Resolved Jing Zhao Actions
        31.
        Disallow write/modify operations on files and directories in a snapshot Sub-task Resolved Brandon Li Actions
        32.
        Complete the code for disallowSnapshot Sub-task Resolved Tsz-wo Sze Actions
        33.
        Update inode in blocksMap when deleting original/snapshot file Sub-task Resolved Jing Zhao Actions
        34.
        FSDirectory#hasSnapshot will NPE when deleting an empty directory Sub-task Resolved Aaron Myers Actions
        35.
        Rename should fail when the destination dir is snapshottable and has snapshots Sub-task Resolved Jing Zhao Actions
        36.
        Add support for scheduled automatic snapshots Sub-task Resolved Suresh Srinivas Actions
        37.
        Add support for restoring/rolling back to a snapshot Sub-task Patch Available Ajith S Actions
        38.
        Add snapshot information to INodesInPath Sub-task Resolved Tsz-wo Sze Actions
        39.
        Additional snapshot tests for more complicated directory structure and modifications. Sub-task Resolved Jing Zhao Actions
        40.
        Add a snapshot parameter to INodeDirectory.getChildrenList() Sub-task Resolved Tsz-wo Sze Actions
        41.
        Add testcases for replication handling in snapshots Sub-task Resolved Jing Zhao Actions
        42.
        Add a snapshot id comparator Sub-task Resolved Tsz-wo Sze Actions
        43.
        rename getter method getMutableX and getXMutable to getXAndEnsureMutable Sub-task Resolved Tsz-wo Sze Actions
        44.
        Support renaming of snapshots Sub-task Resolved Jing Zhao Actions
        45.
        Listing all the current snapshottable directories Sub-task Resolved Jing Zhao Actions
        46.
        Support deleting snapshots Sub-task Resolved Jing Zhao Actions
        47.
        Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer Sub-task Resolved Jing Zhao Actions
        48.
        TestSnapshot fails Sub-task Resolved Jing Zhao Actions
        49.
        Change INode to support HDFS-4103 Sub-task Resolved Tsz-wo Sze Actions
        50.
        Support snapshots up to the snapshot limit Sub-task Resolved Tsz-wo Sze Actions
        51.
        After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode Sub-task Resolved Jing Zhao Actions
        52.
        When listing snapshottable directories, only return those where the user has permission to take snapshots Sub-task Resolved Jing Zhao Actions
        53.
        INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao Actions
        54.
        Combined diff may not be identical to the diffs applied sequentially Sub-task Resolved Tsz-wo Sze Actions
        55.
        In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like Sub-task Resolved Tsz-wo Sze Actions
        56.
        Add support for getting snapshot diff from DistributedFileSystem Sub-task Resolved Jing Zhao Actions
        57.
        Add unit tests for taking snapshots while file appending Sub-task Resolved Jing Zhao Actions
        58.
        Support snapshot in OfflineImageViewer Sub-task Resolved Jing Zhao Actions
        59.
        Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Sub-task Resolved Jing Zhao Actions
        60.
        The snapshot copy INode.recordModification(..) returned is never used Sub-task Resolved Tsz-wo Sze Actions
        61.
        Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class Sub-task Resolved Tsz-wo Sze Actions
        62.
        Support file snapshot with diff lists Sub-task Resolved Tsz-wo Sze Actions
        63.
        Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Sub-task Resolved Tsz-wo Sze Actions
        64.
        Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot Sub-task Resolved Tsz-wo Sze Actions
        65.
        Eliminate the file snapshot circular linked list Sub-task Resolved Tsz-wo Sze Actions
        66.
        Fix snapshot fsimage for file diffs Sub-task Resolved Tsz-wo Sze Actions
        67.
        Fix snapshot diff report Sub-task Resolved Jing Zhao Actions
        68.
        Fix file/directory/snapshot deletion for file diff Sub-task Resolved Jing Zhao Actions
        69.
        Refactor snapshot INode methods Sub-task Resolved Tsz-wo Sze Actions
        70.
        Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot Sub-task Resolved Tsz-wo Sze Actions
        71.
        Update quota verification for snapshot Sub-task Resolved Tsz-wo Sze Actions
        72.
        Add CLI support for snapshot rename, diff report, and snapshottable directory listing Sub-task Resolved Jing Zhao Actions
        73.
        Support listing snapshots under a snapshottable directory using "ls" Sub-task Resolved Jing Zhao Actions
        74.
        Fix INodeFile replacement, TestQuota and javac error Sub-task Resolved Tsz-wo Sze Actions
        75.
        Update SnapshotManager#snapshottables when loading fsimage Sub-task Resolved Jing Zhao Actions
        76.
        Decide the semantic of concat with snapshots Sub-task Resolved Tsz-wo Sze Actions
        77.
        Update rename for snapshots Sub-task Resolved Tsz-wo Sze Actions
        78.
        Snapshot: unprotectedSetReplication may update diskspace count incorrectly Sub-task Resolved Tsz-wo Sze Actions
        79.
        Refactor INodesInPath to a standalone class Sub-task Resolved Tsz-wo Sze Actions
        80.
        Fix FSDirectory#delete when INode#cleanSubtree returns 0 Sub-task Resolved Jing Zhao Actions
        81.
        Update namespace/diskspace after deleting snapshots Sub-task Resolved Jing Zhao Actions
        82.
        Move Diff and EnumCounters to util package Sub-task Resolved Tsz-wo Sze Actions
        83.
        Annotate snapshot tests Sub-task Resolved Arpit Agarwal Actions
        84.
        Restrict snapshot IDs to 24-bits wide Sub-task Resolved Arpit Agarwal Actions
        85.
        Add snapshotdiff and LsSnapshottableDir tools to hdfs script Sub-task Resolved Arpit Agarwal Actions
        86.
        Snapshot: Update FSImage for INodeReference Sub-task Resolved Tsz-wo Sze Actions
        87.
        Not to use INode.getParent() when generating snapshot diff report Sub-task Resolved Jing Zhao Actions
        88.
        Update the FilesDeleted metric while deleting file/dir in the current tree Sub-task Resolved Jing Zhao Actions
        89.
        Fix FSImageFormat#Loader NPE and synchronization issues Sub-task Resolved Jing Zhao Actions
        90.
        Update quota usage when deleting files/dirs that were created after taking the latest snapshot Sub-task Resolved Jing Zhao Actions
        91.
        INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao Actions
        92.
        For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted Sub-task Resolved Jing Zhao Actions
        93.
        Add rename test in TestSnapshot Sub-task Resolved Jing Zhao Actions
        94.
        llegalStateException renaming file under snapshottable directory Sub-task Resolved Arpit Agarwal Actions
        95.
        Disallow users to create file/dir named ".snapshot" Sub-task Resolved Tsz-wo Sze Actions
        96.
        Capture renamed files/directories in snapshot diff report Sub-task Closed Jing Zhao Actions
        97.
        Renaming file in subdirectory of a snapshotted directory does not work. Sub-task Resolved Arpit Agarwal Actions
        98.
        Fix rename across snapshottable directories Sub-task Resolved Jing Zhao Actions
        99.
        Snapshot: Use INode id for image serialization Sub-task Resolved Tsz-wo Sze Actions
        100.
        Fix quota computation for rename with snapshots Sub-task Resolved Jing Zhao Actions
        101.
        Use timestamp as the default snapshot name Sub-task Resolved Tsz-wo Sze Actions
        102.
        Fix the undo section of rename with snapshots Sub-task Resolved Jing Zhao Actions
        103.
        Add a transient flag to file so that transient files won't be included in any snapshot Sub-task Open Tsz-wo Sze Actions
        104.
        disallowSnapshot does not work for root Sub-task Resolved Tsz-wo Sze Actions
        105.
        Fix FilterFileSystem and findbugs warning in Snapshot branch Sub-task Resolved Tsz-wo Sze Actions
        106.
        Add snapshot user guide Sub-task Resolved Tsz-wo Sze Actions
        107.
        Change the parameter type of the snapshot methods in HdfsAdmin to Path Sub-task Resolved Tsz-wo Sze Actions
        108.
        Minor simplifications to snapshot code Sub-task Resolved Arpit Agarwal Actions
        109.
        Fix test failures after merging the mapping from INodeId to INode Sub-task Resolved Jing Zhao Actions
        110.
        Update inodeMap after deleting files/directories/snapshots Sub-task Resolved Jing Zhao Actions
        111.
        Snapshot tests broken after merge from trunk Sub-task Resolved Arpit Agarwal Actions
        112.
        Fix OfflineImageViewer and permission checking for snapshot operations Sub-task Resolved Jing Zhao Actions
        113.
        DisallowSnapshot throws IllegalStateException Sub-task Resolved Jing Zhao Actions
        114.
        Fix javadoc and other warnings in the snapshot branch Sub-task Resolved Tsz-wo Sze Actions
        115.
        Fix appending to a renamed file with snapshot Sub-task Resolved Jing Zhao Actions
        116.
        Convert snapshot user guide to APT from XDOC Sub-task Resolved Aaron Myers Actions
        117.
        Use INodeId to identify the corresponding directory node for FSImage saving/loading Sub-task Resolved Jing Zhao Actions
        118.
        AccessControlException message is changed in snapshot branch Sub-task Resolved Tsz-wo Sze Actions
        119.
        Disallow nested snapshottable directories Sub-task Resolved Tsz-wo Sze Actions
        120.
        snapshotDiff of two invalid snapshots but with same name returns success Sub-task Resolved Jing Zhao Actions
        121.
        Update inodeMap after node replacement Sub-task Resolved Jing Zhao Actions
        122.
        directory is not snapshottable after clrQuota Sub-task Resolved Jing Zhao Actions
        123.
        Fix bugs in quota usage updating/computation Sub-task Resolved Jing Zhao Actions
        124.
        File listing of .snapshot under a non-existing dir throws NullPointer Sub-task Resolved Tsz-wo Sze Actions
        125.
        Update and fix deletion of reference inode Sub-task Resolved Jing Zhao Actions
        126.
        Update computeContentSummary() for the reference nodes in snapshots Sub-task Resolved Tsz-wo Sze Actions
        127.
        Fix INodeDirectoryWithSnapshot#cleanDeletedINode Sub-task Resolved Jing Zhao Actions
        128.
        lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable Sub-task Resolved Jing Zhao Actions
        129.
        Disallowing snapshot on / twice throws error Sub-task Resolved Jing Zhao Actions
        130.
        For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff Sub-task Resolved Jing Zhao Actions
        131.
        Add more unit tests for the undo section of rename Sub-task Resolved Jing Zhao Actions
        132.
        Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree Sub-task Closed Jing Zhao Actions
        133.
        The javadoc of FileWithSnapshot is incorrect Sub-task Closed Tsz-wo Sze Actions
        134.
        Add a test for testing snapshot file length Sub-task Closed Arpit Agarwal Actions
        135.
        Update Snapshot doc for HDFS-4758 Sub-task Closed Tsz-wo Sze Actions
        136.
        Reduce snapshot inode memory usage Sub-task Closed Tsz-wo Sze Actions

          Activity

            People

            • Assignee:
              szetszwo Tsz-wo Sze
              Reporter:
              harip Hari Mankude

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment