Hadoop Common
  1. Hadoop Common
  2. HADOOP-4861

Add disk usage with human-readable size (-duh)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.

      Description

      I think it'd be super useful to have a form of "hadoop dfs -du" that showed the sizes in human-readable format, as in the standard command "du -h".

      1. hadoop-4861.txt
        7 kB
        Todd Lipcon
      2. hadoop-4861.txt
        11 kB
        Todd Lipcon
      3. hadoop-4861.txt
        16 kB
        Todd Lipcon
      4. hadoop-4861.txt
        18 kB
        Todd Lipcon
      5. hadoop-4861.txt
        20 kB
        Todd Lipcon
      6. Test plan for hadoop fs -du -h.html
        8 kB
        Ravi Phulari

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          This patch adds a -h flag to dfs -du and dfs -dus

          It also refactors the two du functions to use the same code path and makes -du -s be synonymous with -dus

          One breaking change is that the output format is now consistent between du and du -s, and it matches the Linux format (size on left, path on right). This may cause incompatibility with users who might parse the output of du, but I think it's worth it for consistency

          Show
          Todd Lipcon added a comment - This patch adds a -h flag to dfs -du and dfs -dus It also refactors the two du functions to use the same code path and makes -du -s be synonymous with -dus One breaking change is that the output format is now consistent between du and du -s, and it matches the Linux format (size on left, path on right). This may cause incompatibility with users who might parse the output of du, but I think it's worth it for consistency
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407208/hadoop-4861.txt
          against trunk revision 772920.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407208/hadoop-4861.txt against trunk revision 772920. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Hudson failure is correct - I'll upload a fixed patch soon.

          Show
          Todd Lipcon added a comment - Hudson failure is correct - I'll upload a fixed patch soon.
          Hide
          Todd Lipcon added a comment -

          Fixes test cases

          Show
          Todd Lipcon added a comment - Fixes test cases
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407845/hadoop-4861.txt
          against trunk revision 774018.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/330/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407845/hadoop-4861.txt against trunk revision 774018. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/330/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Updated patch that should apply cleanly to trunk (previous one was broken by the splitting of the test dirs)

          Show
          Todd Lipcon added a comment - Updated patch that should apply cleanly to trunk (previous one was broken by the splitting of the test dirs)
          Hide
          Todd Lipcon added a comment -

          Slightly updated version of the patch which includes a unit test, and fixes du -h to write "150" for 150 bytes instead of "150.0"

          Show
          Todd Lipcon added a comment - Slightly updated version of the patch which includes a unit test, and fixes du -h to write "150" for 150 bytes instead of "150.0"
          Hide
          Bryan Duxbury added a comment -

          Personally, I don't think backwards compatibility is critical here. I'd much rather have consistency between all the different options.

          I briefly looked over the code and it seems to make sense, though I didn't have time to apply and test it.

          Show
          Bryan Duxbury added a comment - Personally, I don't think backwards compatibility is critical here. I'd much rather have consistency between all the different options. I briefly looked over the code and it seems to make sense, though I didn't have time to apply and test it.
          Hide
          Todd Lipcon added a comment -

          Woops - forgot to include the new data1k file in my previous patch. This patch is identical code, just adds that file.

          Show
          Todd Lipcon added a comment - Woops - forgot to include the new data1k file in my previous patch. This patch is identical code, just adds that file.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12408763/hadoop-4861.txt
          against trunk revision 777761.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 release audit. The applied patch generated 492 release audit warnings (more than the trunk's current 491 warnings).

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/current/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408763/hadoop-4861.txt against trunk revision 777761. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 release audit. The applied patch generated 492 release audit warnings (more than the trunk's current 491 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/current/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Test failures unrelated. The releaseAuditDiffWarnings.txt file linked by the QA bot doesn't load, but I am guessing it is warning that the "data1k" file has no Apache license. This is clearly correct.

          Show
          Todd Lipcon added a comment - Test failures unrelated. The releaseAuditDiffWarnings.txt file linked by the QA bot doesn't load, but I am guessing it is warning that the "data1k" file has no Apache license. This is clearly correct.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Todd!

          Show
          Tom White added a comment - I've just committed this. Thanks Todd!
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21
          Hide
          Ravi Phulari added a comment -

          Hello Todd,
          Wondering why there is no command line help is not added with this patch. Also it did not update user documentation in System shell guide.

          Show
          Ravi Phulari added a comment - Hello Todd, Wondering why there is no command line help is not added with this patch. Also it did not update user documentation in System shell guide.
          Hide
          Todd Lipcon added a comment -

          Hi Ravi,

          Thanks for noticing this. That was an oversight - I'll open another JIRA momentarily against trunk and branch-21 to remedy it.

          -Todd

          Show
          Todd Lipcon added a comment - Hi Ravi, Thanks for noticing this. That was an oversight - I'll open another JIRA momentarily against trunk and branch-21 to remedy it. -Todd

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Bryan Duxbury
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development