Hadoop Common
  1. Hadoop Common
  2. HADOOP-4861

Add disk usage with human-readable size (-duh)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.

      Description

      I think it'd be super useful to have a form of "hadoop dfs -du" that showed the sizes in human-readable format, as in the standard command "du -h".

      1. Test plan for hadoop fs -du -h.html
        8 kB
        Ravi Phulari
      2. hadoop-4861.txt
        7 kB
        Todd Lipcon
      3. hadoop-4861.txt
        11 kB
        Todd Lipcon
      4. hadoop-4861.txt
        16 kB
        Todd Lipcon
      5. hadoop-4861.txt
        18 kB
        Todd Lipcon
      6. hadoop-4861.txt
        20 kB
        Todd Lipcon

        Issue Links

          Activity

          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Ravi Phulari made changes -
          Attachment Test plan for hadoop fs -du -h.html [ 12425414 ]
          Todd Lipcon made changes -
          Link This issue is related to HADOOP-6375 [ HADOOP-6375 ]
          Hide
          Todd Lipcon added a comment -

          Hi Ravi,

          Thanks for noticing this. That was an oversight - I'll open another JIRA momentarily against trunk and branch-21 to remedy it.

          -Todd

          Show
          Todd Lipcon added a comment - Hi Ravi, Thanks for noticing this. That was an oversight - I'll open another JIRA momentarily against trunk and branch-21 to remedy it. -Todd
          Hide
          Ravi Phulari added a comment -

          Hello Todd,
          Wondering why there is no command line help is not added with this patch. Also it did not update user documentation in System shell guide.

          Show
          Ravi Phulari added a comment - Hello Todd, Wondering why there is no command line help is not added with this patch. Also it did not update user documentation in System shell guide.
          Robert Chansler made changes -
          Release Note The output format of hadoop fs -dus is now consistent with the format of hadoop fs -du and with Linux du. The size has moved from the right column to the left column. Users who previously parsed this output should update their scripts.

          New feature: hadoop fs -du -h may now be used for human readable output.
          Output of hadoop fs -dus changed to be consistent with hadoop fs -du and with Linux du. Users who previously parsed this output should update their scripts. New feature hadoop fs -du -h may be used for human readable output.
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21
          Todd Lipcon made changes -
          Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
          Release Note The output format of hadoop fs -dus is now consistent with the format of hadoop fs -du and with Linux du. The size has moved from the right column to the left column. Users who previously parsed this output should update their scripts.

          New feature: hadoop fs -du -h may now be used for human readable output.
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Todd!

          Show
          Tom White added a comment - I've just committed this. Thanks Todd!
          Hide
          Todd Lipcon added a comment -

          Test failures unrelated. The releaseAuditDiffWarnings.txt file linked by the QA bot doesn't load, but I am guessing it is warning that the "data1k" file has no Apache license. This is clearly correct.

          Show
          Todd Lipcon added a comment - Test failures unrelated. The releaseAuditDiffWarnings.txt file linked by the QA bot doesn't load, but I am guessing it is warning that the "data1k" file has no Apache license. This is clearly correct.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12408763/hadoop-4861.txt
          against trunk revision 777761.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 23 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 release audit. The applied patch generated 492 release audit warnings (more than the trunk's current 491 warnings).

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/current/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408763/hadoop-4861.txt against trunk revision 777761. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 release audit. The applied patch generated 492 release audit warnings (more than the trunk's current 491 warnings). +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/current/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/389/console This message is automatically generated.
          Todd Lipcon made changes -
          Attachment hadoop-4861.txt [ 12408763 ]
          Hide
          Todd Lipcon added a comment -

          Woops - forgot to include the new data1k file in my previous patch. This patch is identical code, just adds that file.

          Show
          Todd Lipcon added a comment - Woops - forgot to include the new data1k file in my previous patch. This patch is identical code, just adds that file.
          Hide
          Bryan Duxbury added a comment -

          Personally, I don't think backwards compatibility is critical here. I'd much rather have consistency between all the different options.

          I briefly looked over the code and it seems to make sense, though I didn't have time to apply and test it.

          Show
          Bryan Duxbury added a comment - Personally, I don't think backwards compatibility is critical here. I'd much rather have consistency between all the different options. I briefly looked over the code and it seems to make sense, though I didn't have time to apply and test it.
          Todd Lipcon made changes -
          Attachment hadoop-4861.txt [ 12408735 ]
          Hide
          Todd Lipcon added a comment -

          Slightly updated version of the patch which includes a unit test, and fixes du -h to write "150" for 150 bytes instead of "150.0"

          Show
          Todd Lipcon added a comment - Slightly updated version of the patch which includes a unit test, and fixes du -h to write "150" for 150 bytes instead of "150.0"
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Attachment hadoop-4861.txt [ 12408730 ]
          Hide
          Todd Lipcon added a comment -

          Updated patch that should apply cleanly to trunk (previous one was broken by the splitting of the test dirs)

          Show
          Todd Lipcon added a comment - Updated patch that should apply cleanly to trunk (previous one was broken by the splitting of the test dirs)
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407845/hadoop-4861.txt
          against trunk revision 774018.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/330/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407845/hadoop-4861.txt against trunk revision 774018. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/330/console This message is automatically generated.
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Todd Lipcon [ tlipcon ]
          Todd Lipcon made changes -
          Attachment hadoop-4861.txt [ 12407845 ]
          Hide
          Todd Lipcon added a comment -

          Fixes test cases

          Show
          Todd Lipcon added a comment - Fixes test cases
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Todd Lipcon added a comment -

          Hudson failure is correct - I'll upload a fixed patch soon.

          Show
          Todd Lipcon added a comment - Hudson failure is correct - I'll upload a fixed patch soon.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12407208/hadoop-4861.txt
          against trunk revision 772920.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407208/hadoop-4861.txt against trunk revision 772920. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/300/console This message is automatically generated.
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Attachment hadoop-4861.txt [ 12407208 ]
          Hide
          Todd Lipcon added a comment -

          This patch adds a -h flag to dfs -du and dfs -dus

          It also refactors the two du functions to use the same code path and makes -du -s be synonymous with -dus

          One breaking change is that the output format is now consistent between du and du -s, and it matches the Linux format (size on left, path on right). This may cause incompatibility with users who might parse the output of du, but I think it's worth it for consistency

          Show
          Todd Lipcon added a comment - This patch adds a -h flag to dfs -du and dfs -dus It also refactors the two du functions to use the same code path and makes -du -s be synonymous with -dus One breaking change is that the output format is now consistent between du and du -s, and it matches the Linux format (size on left, path on right). This may cause incompatibility with users who might parse the output of du, but I think it's worth it for consistency
          Bryan Duxbury created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Bryan Duxbury
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development