[HDFS-15492] Make trash root inside each snapshottable directory

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.4.0
• Component/s: hdfs, hdfs-client
• Labels: pull-request-available
• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Siyao Meng Assign to me
• Reporter: Siyao Meng
• Created: 23/Jul/20 17:27
• Updated: 27/Jan/24 03:16
• Resolved: 11/Aug/20 15:53

Description

We have seen FSImage corruption cases (e.g. HDFS-13101) where files inside one snapshottable directories are moved outside of it. The most common case of this is when trash is enabled and user deletes some file via the command line without skipTrash.

This jira aims to make a trash root for each snapshottable directory, same as how encryption zone behaves at the moment.

This will make trash cleanup a little bit more expensive on the NameNode as it will be to iterate all trash roots. But should be fine as long as there aren't many snapshottable directories.

I could make this improvement as an option and disable it by default if needed, such as dfs.namenode.snapshot.trashroot.enabled

One small caveat though, when disabling (disallowing) snapshot on the snapshottable directory when this improvement is in place. The client should merge the snapshottable directory's trash with that user's trash to ensure proper trash cleanup.

Issue Links

• causes: HDFS-15607 Create trash dir when allowing snapshottable dir
• is related to: HDFS-15539 When disallowing snapshot on a dir, throw exception if its trash root is not empty
• is related to: HDFS-15513 Allow client to query snapshot status on one directory
• relates to: HADOOP-8689 Make trash a server side configuration option
• links to: GitHub Pull Request #2176