This patch adds a new column to the output of "hadoop fs -du" and "hadoop fs -dus" which shows the disk space consumed (file size * per-file replication factor) of the paths matched by these commands.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449749/show-space-consumed.txt
against trunk revision 964993.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/console

This message is automatically generated.

Hey Aaron,

Patch looks good. Mind creating o.a.h.fs.TestFsShell.java and adding a test that shows files with two different replication levels works? (might need to mock up the replication level). Also, please test that TestShell and TestHDFSCLI in HDFS still pass for sanity.

Wrt to rationale I think this change is kosher since FileStatus#getReplication is not hdfs-specific.

Marking the jira as an incompatible change since IIRC the FsShell is considered a public API.

Thanks,
Eli

We already have "fs -count <path>" which counts bytes including replications. Is it good enough?

Canceling patch until Nicholas' comments are addressed.

We already have "fs -count <path>" which counts bytes including replications. Is it good enough?

"fs -count <path>" is indeed sufficient. Feel free to close this ticket.

Closing this. Thanks.

Why do we have separate "dus" and "count" commands? They seem to duplicate each other. Since the CLI needs to be backwards compatible not suggesting we remove one, but perhaps we should update the hdfs_shell docs to make it clear that they display the same info if there are not significant differences.

~ $ hadoop fs -dus /user/eli
hdfs://haus01.sf.cloudera.com:10020/user/eli	86183666860

~ $ hadoop fs -count /user/eli
           7           51        86183666860 hdfs://haus01.sf.cloudera.com:10020/user/eli
~ $ 

"fs -count" was introduced for counting name objects. The disk space column was added later on.

A little confused. I thought "fs -count" shows hdfs usage same as "fs -du" at the thrid column.

[knoguchi ~]$ hadoop dfs -dus /user/knoguchi
hdfs://abc-nn1.com/user/knoguchi       2603203340273
[knoguchi ~]$ hadoop dfs -count /user/knoguchi
        1580        20624      2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$ 

If quota is enabled on that dir and "-q" is passed, it would show the remaining raw space available.

[knoguchi ~]$ hadoop dfs -count -q /user/knoguchi
       50000           27796  13194139533312   5384528402193         1580        20624      2603203340273 hdfs://abc-nn1.com/user/knoguchi
[knoguchi ~]$ 

You can get the raw space usage then. (quota - raw_remaining).
However this is only if you have quota enabled on that particular dir.

I think this number(raw usage) would be helpful. Not sure whether this should be in -du or -count and by default or as an option.

Is there an easy cli method of obtaining this information?

If not then this can be a welcome feature in several use cases (Like file server based on HDFS)

I agree with Koji, seeing raw disk space usage should be easy to get from the CLI, you shouldn't have to enable a quota on a directory to see it, eg see below that nothing indicates raw disk usage:

~ $ hadoop fs -mkdir dir
~ $ hadoop fs -put f3mb dir
~ $ hadoop fs -dus dir
hdfs://haus01.sf.cloudera.com:10020/user/eli/dir	3145728
~ $ hadoop fs -count -q dir
        none             inf            none             inf            1            1            3145728 hdfs://haus01.sf.cloudera.com:10020/user/eli/dir

It also sounds like the original issue here was that disk space usage should not have been added to count if it was already available in dus. Even if we added raw disk usage to "count", it's wonky that a user can logical disk usage with "du" but has to switch over to a separate "count" command to get raw disk usage. Why not make all disk usage available from a single command?

Sound reasonable Nicholas?

Updated Aaron T. Myers's patch to patch on top of trunk.
I'd like to point out that creating a snapshot of a directory will increase the disk usage number reported by -du (-s). Is this what we want?

Byron, could you please clarify how -du reporting is different in the presence of snapshots?

In the case when a directory /D and snapshot S are in the exact same state (e.g. a fresh snapshot has been made), everything works fine, meaning the sum of the disk consumed numbers reported by -du /D equals the disk consumed number reported by -du -s /D.
When /D and S start deviating (files getting renamed, deleted, etc.), the disk consumed calculation will take the lastFileSize within the snapshots, find the maximum replication factor for that file within the snapshots, multiply the 2 together, and increment disk consumed by that number, which inflates the total disk consumed calculation, so -du -s /D > the sum of numbers in -du /D.

I'd also like to point out that this implementation only takes replication factor of a file into account, even if that replication factor is greater than number of data nodes, which further inflates the -du calculation. For example, if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10, and report that number.

Byron, Could you post some examples (printouts) in support of your use case.

if we setrep 10 a file when we only have 3 datanodes, -du will still multiply fileLength * 10

This seems to be consistent with how quotas are calculated. DU should report numbers consistent with the quotas. We can change diskSpaceConsumed() if people want more precise usage info both for du and quotas, but at this point it will be a change in semantics. Let's keep it the current way.

Scenario 1: "/test" is a snapshottable directory with a file "a" that has 41 bytes, replication factor 3.
We run hadoop fs -du /test:

41  123  /test/a

which is consistent with what we get when we run hadoop fs -du -s /test:

41  123  /test

When we create a snapshot "ss1" and rerun the -du commands again, we still get the same results as seen above.

Let's say we now run hadoop fs -mv /test/a /test/b.
Now, when we run hadoop fs -du /test, we get:

41  123  /test/b

which is inconsistent with what we see when we run hadoop fs -du -s /test:

41  246  /test

If we report this process again (i.e. create snapshot, rename /test/b to /test/a), we get more and more deviations between the 2 commands.

Scenario 2: we still have "snapshottable directory "/test" with same file "a". We then create a fresh snapshot "ss1". We then run hadoop fs -rm -skipTrash /test/a.
hadoop fs -du /test gives an empty output, as expected.
hadoop fs -du -s /test outputs:

41  123  /test

which makes sense, given that we know about the existence of the snapshot.
However, when we run hadoop fs -du -s /test/.snapshot/ss1, we get:

0  0  /test/.snapshot/ss1

This is inconsistent with the numbers we get when we run hadoop fs -du /test/.snapshot/ss1:

41  123  /test/.snapshot/ss1/a

Upon further investigation, we see that running hadoop fs -du -s /test/.snapshot/anySnapshot gives us the information about the current state of the real directory. This means that hadoop fs -du -s /test/.snapshot/anySnapshot is equivalent to running hadoop fs -du /test/ and adding the numbers up, which is non-intuitive.
For example, let's add a 2 byte, 3 replication file "/test/1" (/test/a is still deleted). Now hadoop fs -du -s /test/.snapshot/ss1 gives us:

2  6  /test/.snapshot/ss1

whereas the results of hadoop fs -du /test/.snapshot/ss1 remains the same:

41  123  /test/.snapshot/ss1/a

Scenario 1 looks rather unusual, but again this is how quotas are counted: When you modify a file in a snapshot, then it is counted twice for quotas. Even if it doesn't actually store axtra blocks. If people want to change this, it should be a different jira related to qoutas. I don't think anything needs to be fixed in this regard here.
Scenario 2 indeed gives inconsistent results for du and dus, because du resolves path to point to the current tree rather than to the snapshot contents. This doesn't look right to me.

Attached new patch.
This patch addresses Scenario 2. Snapshot$Root should computeContentSummary based on its snapshotId rather than the current state. Added unit test to verify this case.

This looks good. +1 from me.
The test indeed failes on trunk and succeeds with the patch.
Let's check with Jenkins.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12676380/HADOOP-6857.patch
against trunk revision db45f04.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.web.TestWebHDFSForHA
org.apache.hadoop.cli.TestHDFSCLI

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4942//console

This message is automatically generated.

Looks like format change of du reporting broke TestHDFSCLI.
Not sure about TestWebHDFSForHA.

Added new patch to fix TestHDFSCLI.
I don't think the failure in TestWebHDFSFOrHA has to do with my changes. It passed locally for me.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
against trunk revision e2be333.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4948//console

This message is automatically generated.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12676965/HADOOP-6857.patch
against trunk revision a52eb4b.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4950//console

This message is automatically generated.

FAILURE: Integrated in Hadoop-trunk-Commit #6345 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6345/)
HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
• hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java

I just committed this. Congratulations Byron.

SUCCESS: Integrated in Hadoop-Yarn-trunk #724 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/724/)
HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1913 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1913/)
HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1938 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1938/)
HADOOP-6857. FsShell should report raw disk usage including replication factor. Contributed by Byron Wong. (shv: rev 28051e415591b8e33dbe954f65230ede23b11683)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
• hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
• hadoop-common-project/hadoop-common/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

This change is incompatible since it changes the output of the CLI command. (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI)
Hi Konstantin Shvachko, should we revert this change from branch-2?

We still need to do either of the below things

• Move the section in CHANGES.txt to incompatible change and keep the fix in branch-2.
• Revert the commit from branch-2.

I like the idea of reverting the change.

Prepared a patch to revert the commit. I'd like to revert it in this weekend if there is at least one +1 or no comment. Any comments are welcome.

Sorry missed your comment 2 days ago. Let me look at it.

So there is a bug fix here and the output format change. The latter is incompatible for branch 2. I think we should revert the format change, but leave the bug fix. Byron, could you please confirm. Akira, does it make sense to you?

Make sense. We should leave the bug fix.

Yea, I agree. The bug fix should remain, but the output changes can be reverted.

Attached HADOOP-6857-revert.patch.
Reverted only the changes that involved command line output.

Thanks Byron Wong for the patch. By the way, can we revert the incompatible change in a separate jira?
Release note automatically collects information from JIRA (http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/releasenotes.html), so I'd like to make them as follows:

• HADOOP-6857: Fix version -> 3.0.0 and mark as incompatible change.
• A separate jira that reverts incompatible change: Fix version -> 2.7.0 (commit branch-2 only)

Filed HADOOP-11666.

Moved the section to 3.0.0 in CHANGES.txt. Thanks Byron Wong and Konstantin Shvachko!

FAILURE: Integrated in Hadoop-trunk-Commit #7251 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7251/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/856/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/)
Move HADOOP-6857 to 3.0.0. (aajisaka: rev 29bb6898654199a809f1c3e8e536a63fb0d4f073)

• hadoop-common-project/hadoop-common/CHANGES.txt

This is still sitting in the changes.txt file for branch-2 and probably 2.7....

Because it was committed to branch-2 and 2.7. As discussed earlier there is bug fix, which went everywhere, and the print format change, which was reverted from branch 2 by HADOOP-11666.