Details
-
Type:
New Feature
-
Status: Closed
-
Priority:
Trivial
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.21.0
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Incompatible change, Reviewed
-
Release Note:Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.
Description
I think it'd be super useful to have a form of "hadoop dfs -du" that showed the sizes in human-readable format, as in the standard command "du -h".
Issue Links
- is related to
-
HADOOP-6375
Update documentation for FsShell du command
-
- Closed
-
Activity
| Status | Resolved [ 5 ] | Closed [ 6 ] |
| Attachment | Test plan for hadoop fs -du -h.html [ 12425414 ] |
| Link |
This issue is related to |
Hello Todd,
Wondering why there is no command line help is not added with this patch. Also it did not update user documentation in System shell guide.
| Release Note |
The output format of hadoop fs -dus is now consistent with the format of hadoop fs -du and with Linux du. The size has moved from the right column to the left column. Users who previously parsed this output should update their scripts.
New feature: hadoop fs -du -h may now be used for human readable output. |
Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output. |
Editorial pass over all release notes prior to publication of 0.21
| Hadoop Flags | [Reviewed] | [Incompatible change, Reviewed] |
| Release Note |
The output format of hadoop fs -dus is now consistent with the format of hadoop fs -du and with Linux du. The size has moved from the right column to the left column. Users who previously parsed this output should update their scripts.
New feature: hadoop fs -du -h may now be used for human readable output. |
| Status | Patch Available [ 10002 ] | Resolved [ 5 ] |
| Hadoop Flags | [Reviewed] | |
| Fix Version/s | 0.21.0 [ 12313563 ] | |
| Resolution | Fixed [ 1 ] |
Test failures unrelated. The releaseAuditDiffWarnings.txt file linked by the QA bot doesn't load, but I am guessing it is warning that the "data1k" file has no Apache license. This is clearly correct.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408763/hadoop-4861.txt
against trunk revision 777761.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 23 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 release audit. The applied patch generated 492 release audit warnings (more than the trunk's current 491 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/console
This message is automatically generated.
| Attachment | hadoop-4861.txt [ 12408763 ] |
Woops - forgot to include the new data1k file in my previous patch. This patch is identical code, just adds that file.
Personally, I don't think backwards compatibility is critical here. I'd much rather have consistency between all the different options.
I briefly looked over the code and it seems to make sense, though I didn't have time to apply and test it.
| Attachment | hadoop-4861.txt [ 12408735 ] |
Slightly updated version of the patch which includes a unit test, and fixes du -h to write "150" for 150 bytes instead of "150.0"
| Status | Open [ 1 ] | Patch Available [ 10002 ] |
| Attachment | hadoop-4861.txt [ 12408730 ] |
Updated patch that should apply cleanly to trunk (previous one was broken by the splitting of the test dirs)
| Status | Patch Available [ 10002 ] | Open [ 1 ] |
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407845/hadoop-4861.txt
against trunk revision 774018.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/330/console
This message is automatically generated.
| Status | Open [ 1 ] | Patch Available [ 10002 ] |
| Assignee | Todd Lipcon [ tlipcon ] |
| Attachment | hadoop-4861.txt [ 12407845 ] |
Fixes test cases
| Status | Patch Available [ 10002 ] | Open [ 1 ] |
Hudson failure is correct - I'll upload a fixed patch soon.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407208/hadoop-4861.txt
against trunk revision 772920.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/console
This message is automatically generated.
| Status | Open [ 1 ] | Patch Available [ 10002 ] |
| Field | Original Value | New Value |
|---|---|---|
| Attachment | hadoop-4861.txt [ 12407208 ] |
This patch adds a -h flag to dfs -du and dfs -dus
It also refactors the two du functions to use the same code path and makes -du -s be synonymous with -dus
One breaking change is that the output format is now consistent between du and du -s, and it matches the Linux format (size on left, path on right). This may cause incompatibility with users who might parse the output of du, but I think it's worth it for consistency
Hi Ravi,
Thanks for noticing this. That was an oversight - I'll open another JIRA momentarily against trunk and branch-21 to remedy it.
-Todd