[HDFS-15477] Enforce ordered snapshot deletion

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: snapshots
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 18/Jul/20 19:04
• Updated: 14/Aug/20 17:37

Description

Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose a configurable feature for enforcing ordered snapshot deletion – only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted.

The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot.

One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted.

Issue Links

• is related to: HDFS-15516 Add info for create flags in NameNode audit logs
• is related to: HDFS-15513 Allow client to query snapshot status on one directory
• is related to: HDFS-15519 Check inaccessible INodes in FsImageValidation

Sub-Tasks

• Ordered snapshot deletion: make it a configurable feature: Resolved
• Ordered snapshot deletion: record snapshot deletion in XAttr: Resolved
• Ordered snapshot deletion: garbage collect deleted snapshots: Resolved
• Add a command to list all snapshots for a snaphottable root with snapshot Ids: Resolved
• Show snapshots deletion status in snapList cmd: Resolved
• Ordered snapshot deletion: Disallow rename between two snapshottable directories: Resolved
• Address checkstyle issues reported with HDFS-15480: Resolved
• Make snapshot limit on global as well per snapshot root directory configurable: Resolved
• Make trash root inside each snapshottable directory: Resolved
• Create trash dir when allowing snapshottable dir: Resolved
• Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir: Resolved
• When disallowing snapshot on a dir, throw exception if its trash root is not empty: Resolved
• Disallow making a Snapshottable directory unsnapshottable if it has no empty snapshot trash directory: Resolved
• allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check: Resolved
• Make trash root inside each snapshottable directory for WebHDFS: Resolved
• Initialize snapshot trash root during NameNode startup if enabled: Resolved
• Add UI for deleted snapshots: Resolved
• In-order deletion of snapshots: Diff lists must be update only in the last snapshot: Resolved
• Wrong operation name in FsNamesystem for listSnapshots: Resolved
• Add edit log entry for Snapshot deletion GC thread snapshot deletion: Resolved
• Namenode UI: List snapshots in separate table for each snapshottable directory: Resolved
• Add identified snapshot corruption tests for ordered snapshot deletion: Resolved
• namenode start failed to start when dfs.namenode.snapshot.max.limit set: Resolved
• namenode fails to start when ordered snapshot deletion feature is disabled: Resolved
• Add list Snapshot command in WebHDFS: Resolved
• allowSnapshot fails when directory already has a Trash sub directory: Resolved
• Rename snapshots while marking them deleted: Resolved
• Ensure snapshot root trash provisioning happens only post safe mode exit: Resolved
• standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories: Resolved
• Implement dfsadmin -provisionSnapshotTrash -all: Resolved
• Metric for ordered snapshot deletion GC thread: Open
• WebHDFS: Support provisionSnapshotTrash: Open
• Update Apache documentation for new ordered snapshot deletion feature: Open