[HDFS-8986] Add option to -du to calculate directory space usage excluding snapshots

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: snapshots
• Labels: supportability
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Release Note: Add a -x option for "hdfs -du" and "hdfs -count" commands to exclude snapshots from being calculated.
• Assignee: Xiao Chen
• Reporter: Gautam Gopalakrishnan
• Created: 28/Aug/15 03:42
• Updated: 02/Oct/19 17:15
• Resolved: 24/Aug/16 16:26

Description

When running hadoop fs -du on a snapshotted directory (or one of its children), the report includes space consumed by blocks that are only present in the snapshots. This is confusing for end users.

$  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
799.7 M  2.3 G  /tmp/parent
799.7 M  2.3 G  /tmp/parent/sub1

$ hdfs dfs -createSnapshot /tmp/parent snap1
Created snapshot /tmp/parent/.snapshot/snap1

$ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
...

$ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
799.7 M  2.3 G  /tmp/parent
799.7 M  2.3 G  /tmp/parent/sub1

$ hdfs dfs -deleteSnapshot /tmp/parent snap1
$ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
0  0  /tmp/parent
0  0  /tmp/parent/sub1

It would be helpful if we had a flag, say -X, to exclude any snapshot related disk usage in the output

Issue Links

• is related to: HDFS-10875 Optimize du -x to cache intermediate result