[HDFS-2802] Support for RW/RO snapshots in HDFS

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: namenode
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Incompatible change
• Assignee: Tsz-wo Sze
• Reporter: Hari Mankude
• Created: 17/Jan/12 18:47
• Updated: 02/Jul/17 00:23
• Resolved: 24/Feb/14 21:56

Description

Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

Issue Links

• duplicates: HDFS-233 Support for snapshots
• incorporates: HDFS-4578 Restrict snapshot IDs to 24-bits wide
• is related to: HDFS-4107 Add utility methods to cast INode to INodeFile and INodeFileUnderConstruction
• is related to: HDFS-4112 A few improvements on INodeDirectory
• is related to: HDFS-4209 Clean up the addNode/addChild/addChildNoQuotaCheck methods in FSDirectory
• is related to: HDFS-4826 TestNestedSnapshots times out due to repeated slow edit log flushes when running on virtualized disk
• is related to: HDFS-4213 When the client calls hsync, allows the client to update the file length in the NameNode
• is related to: HDFS-4124 Refactor INodeDirectory#getExistingPathINodes() to enable returning more than INode array
• is related to: HDFS-4037 Rename the getReplication() method in BlockCollection to getBlockReplication()
• is related to: HDFS-4206 Change the fields in INode and its subclasses to private
• is related to: HDFS-4215 Improvements on INode and image loading
• is related to: HDFS-4151 Passing INodesInPath instead of INode[] in FSDirectory
• is related to: HDFS-4152 Add a new class for the parameter in INode.collectSubtreeBlocksAndClear(..)
• is related to: HDFS-4074 Remove empty constructors for INode
• is related to: HDFS-4129 Add utility methods to dump NameNode in memory tree for testing
• relates to: HBASE-6055 Offline Snapshots in HBase 0.96
• relates to: HBASE-50 Snapshot of table

Sub-Tasks

• Snapshot of Being Written Files: Resolved
• Support snapshot of single files: Resolved
• Support snapshottable INodeDirectory: Resolved
• Handle replication in snapshots: Resolved
• Add SnapshotManager: Resolved
• Add editlog opcodes for snapshot create and delete operations: Resolved
• Protocol changes for snapshots: Resolved
• provide CLI support for allow and disallow snapshot on a directory: Resolved
• Add editlog opcodes to allow and disallow snapshots on a directory: Resolved
• Protocol changes for listSnapshots functionality: Resolved
• Add snapshot quota to limit the number of snapshots: Resolved
• Update file deletion logic to support snapshot files: Resolved
• Add snapshot related metrics: Resolved
• Add snapshot information to namenode WebUI: Closed
• provide CLI support for createSnapshot: Resolved
• Support append to original files which are snapshotted: Resolved
• Test replication with snapshots: Resolved
• Support O(1) snapshot creation: Resolved
• Support snapshot of subtrees: Resolved
• Add audit log for snapshot operations: Resolved
• Change INodeDirectory.getExistingPathINodes(..) to work with snapshots: Resolved
• Complete the allowSnapshot code and add a test for it: Resolved
• Use a persistent data structure for snapshots: Resolved
• Add reading/writing snapshot information to FSImage: Resolved
• Add capability to namenode to get snapshot diff: Resolved
• Add testcases for testing basic snapshot functionalities: Resolved
• Directory diff in snapshots: Resolved
• Create test for all snapshot-related metrics: Resolved
• Further work on changing INodeFile.blocks to private in Snapshot branch: Resolved
• Deletion of snapshottable dir with snapshots should fail: Resolved
• Disallow write/modify operations on files and directories in a snapshot: Resolved
• Complete the code for disallowSnapshot: Resolved
• Update inode in blocksMap when deleting original/snapshot file: Resolved
• FSDirectory#hasSnapshot will NPE when deleting an empty directory: Resolved
• Rename should fail when the destination dir is snapshottable and has snapshots: Resolved
• Add support for scheduled automatic snapshots: Resolved
• Add support for restoring/rolling back to a snapshot: Patch Available
• Add snapshot information to INodesInPath: Resolved
• Additional snapshot tests for more complicated directory structure and modifications.: Resolved
• Add a snapshot parameter to INodeDirectory.getChildrenList(): Resolved
• Add testcases for replication handling in snapshots: Resolved
• Add a snapshot id comparator: Resolved
• rename getter method getMutableX and getXMutable to getXAndEnsureMutable: Resolved
• Support renaming of snapshots: Resolved
• Listing all the current snapshottable directories: Resolved
• Support deleting snapshots: Resolved
• Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer: Resolved
• TestSnapshot fails: Resolved
• Change INode to support HDFS-4103: Resolved
• Support snapshots up to the snapshot limit: Resolved
• After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode: Resolved
• When listing snapshottable directories, only return those where the user has permission to take snapshots: Resolved
• INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot: Resolved
• Combined diff may not be identical to the diffs applied sequentially: Resolved
• In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like: Resolved
• Add support for getting snapshot diff from DistributedFileSystem: Resolved
• Add unit tests for taking snapshots while file appending: Resolved
• Support snapshot in OfflineImageViewer: Resolved
• Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading: Resolved
• The snapshot copy INode.recordModification(..) returned is never used: Resolved
• Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class: Resolved
• Support file snapshot with diff lists: Resolved
• Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot: Resolved
• Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot: Resolved
• Eliminate the file snapshot circular linked list: Resolved
• Fix snapshot fsimage for file diffs: Resolved
• Fix snapshot diff report: Resolved
• Fix file/directory/snapshot deletion for file diff: Resolved
• Refactor snapshot INode methods: Resolved
• Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot: Resolved
• Update quota verification for snapshot: Resolved
• Add CLI support for snapshot rename, diff report, and snapshottable directory listing: Resolved
• Support listing snapshots under a snapshottable directory using "ls": Resolved
• Fix INodeFile replacement, TestQuota and javac error: Resolved
• Update SnapshotManager#snapshottables when loading fsimage: Resolved
• Decide the semantic of concat with snapshots: Resolved
• Update rename for snapshots: Resolved
• Snapshot: unprotectedSetReplication may update diskspace count incorrectly: Resolved
• Refactor INodesInPath to a standalone class: Resolved
• Fix FSDirectory#delete when INode#cleanSubtree returns 0: Resolved
• Update namespace/diskspace after deleting snapshots: Resolved
• Move Diff and EnumCounters to util package: Resolved
• Annotate snapshot tests: Resolved
• Restrict snapshot IDs to 24-bits wide: Resolved
• Add snapshotdiff and LsSnapshottableDir tools to hdfs script: Resolved
• Snapshot: Update FSImage for INodeReference: Resolved
• Not to use INode.getParent() when generating snapshot diff report: Resolved
• Update the FilesDeleted metric while deleting file/dir in the current tree: Resolved
• Fix FSImageFormat#Loader NPE and synchronization issues: Resolved
• Update quota usage when deleting files/dirs that were created after taking the latest snapshot: Resolved
• INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot: Resolved
• For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted: Resolved
• Add rename test in TestSnapshot: Resolved
• llegalStateException renaming file under snapshottable directory: Resolved
• Disallow users to create file/dir named ".snapshot": Resolved
• Capture renamed files/directories in snapshot diff report: Closed
• Renaming file in subdirectory of a snapshotted directory does not work.: Resolved
• Fix rename across snapshottable directories: Resolved
• Snapshot: Use INode id for image serialization: Resolved
• Fix quota computation for rename with snapshots: Resolved
• Use timestamp as the default snapshot name: Resolved
• Fix the undo section of rename with snapshots: Resolved
• Add a transient flag to file so that transient files won't be included in any snapshot: Open
• disallowSnapshot does not work for root: Resolved
• Fix FilterFileSystem and findbugs warning in Snapshot branch: Resolved
• Add snapshot user guide: Resolved
• Change the parameter type of the snapshot methods in HdfsAdmin to Path: Resolved
• Minor simplifications to snapshot code: Resolved
• Fix test failures after merging the mapping from INodeId to INode: Resolved
• Update inodeMap after deleting files/directories/snapshots: Resolved
• Snapshot tests broken after merge from trunk: Resolved
• Fix OfflineImageViewer and permission checking for snapshot operations: Resolved
• DisallowSnapshot throws IllegalStateException: Resolved
• Fix javadoc and other warnings in the snapshot branch: Resolved
• Fix appending to a renamed file with snapshot: Resolved
• Convert snapshot user guide to APT from XDOC: Resolved
• Use INodeId to identify the corresponding directory node for FSImage saving/loading: Resolved
• AccessControlException message is changed in snapshot branch: Resolved
• Disallow nested snapshottable directories: Resolved
• snapshotDiff of two invalid snapshots but with same name returns success: Resolved
• Update inodeMap after node replacement: Resolved
• directory is not snapshottable after clrQuota: Resolved
• Fix bugs in quota usage updating/computation: Resolved
• File listing of .snapshot under a non-existing dir throws NullPointer: Resolved
• Update and fix deletion of reference inode: Resolved
• Update computeContentSummary() for the reference nodes in snapshots: Resolved
• Fix INodeDirectoryWithSnapshot#cleanDeletedINode: Resolved
• lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable: Resolved
• Disallowing snapshot on / twice throws error: Resolved
• For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff: Resolved
• Add more unit tests for the undo section of rename: Resolved
• Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree: Closed
• The javadoc of FileWithSnapshot is incorrect: Closed
• Add a test for testing snapshot file length: Closed
• Update Snapshot doc for HDFS-4758: Closed
• Reduce snapshot inode memory usage: Closed