Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11657

Align the output of `hadoop fs -du` to be more Unix-like

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The output of du has now been made more Unix-like, with aligned output.

      Description

      This could be formatted a little nicer:

      $ hdfs dfs -du -s /apps/hbase/data/data/default/*
      22604541341  /apps/hbase/data/data/default/IntegrationTestBulkLoad
      896656491  /apps/hbase/data/data/default/IntegrationTestIngest
      33776145312  /apps/hbase/data/data/default/IntegrationTestLoadAndVerify
      83512463  /apps/hbase/data/data/default/SendTracesTable
      532898  /apps/hbase/data/data/default/TestAcidGuarantees
      27294  /apps/hbase/data/data/default/demo_table
      1410  /apps/hbase/data/data/default/example
      2531532801  /apps/hbase/data/data/default/loadtest_d1
      901  /apps/hbase/data/data/default/table_qho71mpvj8
      1433  /apps/hbase/data/data/default/tcreatetbl
      1690  /apps/hbase/data/data/default/tdelrowtbl
      360  /apps/hbase/data/data/default/testtbl1
      360  /apps/hbase/data/data/default/testtbl2
      360  /apps/hbase/data/data/default/testtbl3
      1515  /apps/hbase/data/data/default/tquerytbl
      1513  /apps/hbase/data/data/default/tscantbl
      
      1. HDFS-5292.patch
        4 kB
        Akira Ajisaka
      2. HDFS-5292-002.patch
        3 kB
        Akira Ajisaka

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        Hi, Nick Dimiduk. Do you have any specific ideas in mind on what you'd like to see in the output? More Unix-like du options like -h for human-readable and -c for a total? Something else? Thanks!

        Show
        cnauroth Chris Nauroth added a comment - Hi, Nick Dimiduk . Do you have any specific ideas in mind on what you'd like to see in the output? More Unix-like du options like -h for human-readable and -c for a total? Something else? Thanks!
        Hide
        ndimiduk Nick Dimiduk added a comment -

        The jagged output is what caught my attention as it differs from the unix-like rendering.

        $ du -s ./*
        768	./CHANGES.txt
        8	./HDP-CHANGES.txt
        24	./LICENSE.txt
        8	./NOTICE.txt
        8	./README.txt
        328	./bin
        64	./conf
        424	./dev-support
        176	./hbase-assembly
        14184	./hbase-client
        8112	./hbase-common
        6368	./hbase-examples
        1000	./hbase-hadoop-compat
        1616	./hbase-hadoop1-compat
        1664	./hbase-hadoop2-compat
        2120	./hbase-hbase-master-hor17n34.gq1.ygridcore.net.log.gz
        3104	./hbase-it
        2632	./hbase-prefix-tree
        46904	./hbase-protocol
        88432	./hbase-server
        1344	./hbase-shell
        16096	./hbase-thrift
        2872	./logs
        152	./pom.xml
        10184	./small1-run-hdp2.0.tgz
        10480	./small2-run-hdp1.3.tgz
        76328	./small2.tgz
        1832	./src
        80	./target
        
        Show
        ndimiduk Nick Dimiduk added a comment - The jagged output is what caught my attention as it differs from the unix-like rendering. $ du -s ./* 768 ./CHANGES.txt 8 ./HDP-CHANGES.txt 24 ./LICENSE.txt 8 ./NOTICE.txt 8 ./README.txt 328 ./bin 64 ./conf 424 ./dev-support 176 ./hbase-assembly 14184 ./hbase-client 8112 ./hbase-common 6368 ./hbase-examples 1000 ./hbase-hadoop-compat 1616 ./hbase-hadoop1-compat 1664 ./hbase-hadoop2-compat 2120 ./hbase-hbase-master-hor17n34.gq1.ygridcore.net.log.gz 3104 ./hbase-it 2632 ./hbase-prefix-tree 46904 ./hbase-protocol 88432 ./hbase-server 1344 ./hbase-shell 16096 ./hbase-thrift 2872 ./logs 152 ./pom.xml 10184 ./small1-run-hdp2.0.tgz 10480 ./small2-run-hdp1.3.tgz 76328 ./small2.tgz 1832 ./src 80 ./target
        Hide
        ajisakaa Akira Ajisaka added a comment -

        It seems the output of `dfs -du` is not properly rendered when input arguments are "multiple". Attaching a patch to fix the issue.

        Show
        ajisakaa Akira Ajisaka added a comment - It seems the output of `dfs -du` is not properly rendered when input arguments are "multiple". Attaching a patch to fix the issue.
        Hide
        sinchii Shinichi Yamashita added a comment -

        +1, LGTM. I confirmed that the output of the command was displayed in UNIX-like.

        Show
        sinchii Shinichi Yamashita added a comment - +1, LGTM. I confirmed that the output of the command was displayed in UNIX-like.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12607471/HDFS-5292.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5896//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5896//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12607471/HDFS-5292.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5896//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5896//console This message is automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Rebased for trunk.

        Show
        ajisakaa Akira Ajisaka added a comment - Rebased for trunk.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12699578/HDFS-5292-002.patch
        against trunk revision 9a3e292.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9616//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9616//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699578/HDFS-5292-002.patch against trunk revision 9a3e292. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9616//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9616//console This message is automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        The test failure looks unrelated to the patch.

        Show
        ajisakaa Akira Ajisaka added a comment - The test failure looks unrelated to the patch.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        The failure was reported at HDFS-7813 and fixed.

        Show
        ajisakaa Akira Ajisaka added a comment - The failure was reported at HDFS-7813 and fixed.
        Hide
        aw Allen Wittenauer added a comment -

        +1, but for trunk only since it changes the output in an incompatible way.

        Show
        aw Allen Wittenauer added a comment - +1, but for trunk only since it changes the output in an incompatible way.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks Allen Wittenauer for the review.

        but for trunk only since it changes the output in an incompatible way.

        Make sense to me. Regarding the output of du, I'm thinking HADOOP-6857 is also incompatible. I'll mark this jira.

        Show
        ajisakaa Akira Ajisaka added a comment - Thanks Allen Wittenauer for the review. but for trunk only since it changes the output in an incompatible way. Make sense to me. Regarding the output of du, I'm thinking HADOOP-6857 is also incompatible. I'll mark this jira.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Moved from HDFS to Common since the change is in the common side.

        Show
        ajisakaa Akira Ajisaka added a comment - Moved from HDFS to Common since the change is in the common side.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk. Thanks Shinichi Yamashita and Allen Wittenauer for review!

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk. Thanks Shinichi Yamashita and Allen Wittenauer for review!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #7232 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7232/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7232 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7232/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12699578/HDFS-5292-002.patch
        against trunk revision e9ac88a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.util.TestByteArrayManager

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5804//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5804//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699578/HDFS-5292-002.patch against trunk revision e9ac88a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.util.TestByteArrayManager Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5804//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5804//console This message is automatically generated.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #120 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/120/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #120 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/120/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #854 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/854/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #854 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/854/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2052 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2052/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2052 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2052/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/111/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/111/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/120/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/120/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2070 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2070/)
        HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2070 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2070/ ) HADOOP-11657 . Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) (aajisaka: rev 30e73ebc77654ff941bcae5b6fb11d52c6d74d2e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java hadoop-common-project/hadoop-common/CHANGES.txt

          People

          • Assignee:
            ajisakaa Akira Ajisaka
            Reporter:
            ndimiduk Nick Dimiduk
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development