Hadoop Common
  1. Hadoop Common
  2. HADOOP-3459

Change dfs -ls listing to closely match format on Linux

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed the output of the "fs -ls" command to more closely match familiar Linux format. Applications that parse the command output should be reviewed.

      Description

      Change the dfs -ls command output to closely match the output format on Linux

      1. HADOOP-3459.patch
        1 kB
        Mukund Madhugiri
      2. HADOOP-3459.patch
        1 kB
        Mukund Madhugiri

        Activity

        Hide
        Mukund Madhugiri added a comment -

        The current output format (in trunk) is:
        Replication Size Date Permissions Owner Group Size Name

        Example:
        1 196 2008-05-22 19:59 rw-rr- hdfsuser supergroup /conf/commons-logging.properties

        I have a patch that changes it to:

        Permissions Replication Owner Group Size Date Name

        Example:
        rw-rr- 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties

         
        CURRENT:
        1    196         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/commons-logging.properties
             0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf
             0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf2
             0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf3
        1    535         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/configuration.xsl
        1    38550       2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-default.xml
        1    2237        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-env.sh
        1    1245        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-metrics.properties
        1    463         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-site.xml
        1    2724        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/log4j.properties
        1    10          2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/masters
        1    10          2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/slaves
        1    401         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/sslinfo.xml.example
        
        AFTER PATCH:
        -rw-r--r--  1    hdfsuser  supergroup           196  2008-05-22 19:59  /conf/commons-logging.properties
        drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf
        drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf2
        drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf3
        -rw-r--r--  1    hdfsuser  supergroup           535  2008-05-22 19:59  /conf/configuration.xsl
        -rw-r--r--  1    hdfsuser  supergroup         38550  2008-05-22 19:59  /conf/hadoop-default.xml
        -rw-r--r--  1    hdfsuser  supergroup          2237  2008-05-22 19:59  /conf/hadoop-env.sh
        -rw-r--r--  1    hdfsuser  supergroup          1245  2008-05-22 19:59  /conf/hadoop-metrics.properties
        -rw-r--r--  1    hdfsuser  supergroup           463  2008-05-22 19:59  /conf/hadoop-site.xml
        -rw-r--r--  1    hdfsuser  supergroup          2724  2008-05-22 19:59  /conf/log4j.properties
        -rw-r--r--  1    hdfsuser  supergroup            10  2008-05-22 19:59  /conf/masters
        -rw-r--r--  1    hdfsuser  supergroup            10  2008-05-22 19:59  /conf/slaves
        -rw-r--r--  1    hdfsuser  supergroup           401  2008-05-22 19:59  /conf/sslinfo.xml.example
        
        Show
        Mukund Madhugiri added a comment - The current output format (in trunk) is: Replication Size Date Permissions Owner Group Size Name Example: 1 196 2008-05-22 19:59 rw-r r - hdfsuser supergroup /conf/commons-logging.properties I have a patch that changes it to: Permissions Replication Owner Group Size Date Name Example: rw-r r - 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties CURRENT: 1 196 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/commons-logging.properties 0 2008-05-22 19:59 drwxr-xr-x hdfsuser supergroup /conf/conf 0 2008-05-22 19:59 drwxr-xr-x hdfsuser supergroup /conf/conf2 0 2008-05-22 19:59 drwxr-xr-x hdfsuser supergroup /conf/conf3 1 535 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/configuration.xsl 1 38550 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/hadoop-default.xml 1 2237 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/hadoop-env.sh 1 1245 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/hadoop-metrics.properties 1 463 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/hadoop-site.xml 1 2724 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/log4j.properties 1 10 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/masters 1 10 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/slaves 1 401 2008-05-22 19:59 -rw-r--r-- hdfsuser supergroup /conf/sslinfo.xml.example AFTER PATCH: -rw-r--r-- 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf2 drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf3 -rw-r--r-- 1 hdfsuser supergroup 535 2008-05-22 19:59 /conf/configuration.xsl -rw-r--r-- 1 hdfsuser supergroup 38550 2008-05-22 19:59 /conf/hadoop-default.xml -rw-r--r-- 1 hdfsuser supergroup 2237 2008-05-22 19:59 /conf/hadoop-env.sh -rw-r--r-- 1 hdfsuser supergroup 1245 2008-05-22 19:59 /conf/hadoop-metrics.properties -rw-r--r-- 1 hdfsuser supergroup 463 2008-05-22 19:59 /conf/hadoop-site.xml -rw-r--r-- 1 hdfsuser supergroup 2724 2008-05-22 19:59 /conf/log4j.properties -rw-r--r-- 1 hdfsuser supergroup 10 2008-05-22 19:59 /conf/masters -rw-r--r-- 1 hdfsuser supergroup 10 2008-05-22 19:59 /conf/slaves -rw-r--r-- 1 hdfsuser supergroup 401 2008-05-22 19:59 /conf/sslinfo.xml.example
        Hide
        Lohit Vijayarenu added a comment -

        +1 this looks good

        Show
        Lohit Vijayarenu added a comment - +1 this looks good
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382951/HADOOP-3459.patch
        against trunk revision 659405.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382951/HADOOP-3459.patch against trunk revision 659405. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/console This message is automatically generated.
        Hide
        Mukund Madhugiri added a comment -

        Regarding -1 for the tests, I am following up this JIRA was a complete set of tests for DFS commands via HADOOP-3100. The output of -ls and other commands are tested in that JIRA.

        Show
        Mukund Madhugiri added a comment - Regarding -1 for the tests, I am following up this JIRA was a complete set of tests for DFS commands via HADOOP-3100 . The output of -ls and other commands are tested in that JIRA.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        If some replication has two digits, the format won't look good since "-" is a fixed String.

        Show
        Tsz Wo Nicholas Sze added a comment - If some replication has two digits, the format won't look good since "-" is a fixed String.
        Hide
        Mukund Madhugiri added a comment -

        Nicholas,
        Good point. I could right justify it, so it does not look bad. Any other suggestions welcome or I will go ahead and upload a new patch which right justifies the replication factor, along the lines of the file size

        -rw-r--r--      1  hdfsuser  supergroup           196  2008-05-22 19:59  /conf/commons-logging.properties
        drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf
        drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf2
        drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf3
        -rw-r--r--     10  hdfsuser  supergroup           535  2008-05-22 19:59  /conf/configuration.xsl
        
        Show
        Mukund Madhugiri added a comment - Nicholas, Good point. I could right justify it, so it does not look bad. Any other suggestions welcome or I will go ahead and upload a new patch which right justifies the replication factor, along the lines of the file size -rw-r--r-- 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf2 drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf3 -rw-r--r-- 10 hdfsuser supergroup 535 2008-05-22 19:59 /conf/configuration.xsl
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The new output looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - The new output looks good.
        Hide
        Owen O'Malley added a comment -

        I really think it is required to have a fixed space between the fields. So I would much rather have spaces included in the format string than extra width.

               System.out.printf("%-"+ maxReplication + "s ", 
                  (!stat.isDir() ? stat.getReplication() : "-"));
        

        instead of

               System.out.printf("%-"+ (maxReplication + BORDER) +
                 "s", (!stat.isDir() ? stat.getReplication() : "-"));
        
        Show
        Owen O'Malley added a comment - I really think it is required to have a fixed space between the fields. So I would much rather have spaces included in the format string than extra width. System .out.printf( "%-" + maxReplication + "s " , (!stat.isDir() ? stat.getReplication() : "-" )); instead of System .out.printf( "%-" + (maxReplication + BORDER) + "s" , (!stat.isDir() ? stat.getReplication() : "-" ));
        Hide
        Owen O'Malley added a comment -

        I guess, I should have explained better. The problem that I've observed in the past, is that if the field is too wide, you lose the separator between the fields, making it impossible to figure out what is going wrong.

        Show
        Owen O'Malley added a comment - I guess, I should have explained better. The problem that I've observed in the past, is that if the field is too wide, you lose the separator between the fields, making it impossible to figure out what is going wrong.
        Hide
        Mukund Madhugiri added a comment -

        Updated patch based on feedback from Owen. Output looks like this:

        -rw-r--r--   1 hdfsuser supergroup        196 2008-05-22 19:59 /conf/commons-logging.properties
        drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf
        drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf2
        drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf3
        -rw-r--r--  10 hdfsuser supergroup        535 2008-05-22 19:59 /conf/configuration.xsl
        -rw-r--r--   1 hdfsuser supergroup      38550 2008-05-22 19:59 /conf/hadoop-default.xml
        -rw-r--r--   1 hdfsuser supergroup       2237 2008-05-22 19:59 /conf/hadoop-env.sh
        -rw-r--r--   1 hdfsuser supergroup       1245 2008-05-22 19:59 /conf/hadoop-metrics.properties
        -rw-r--r--   1 hdfsuser supergroup        463 2008-05-22 19:59 /conf/hadoop-site.xml
        -rw-r--r--   1 hdfsuser supergroup       2724 2008-05-22 19:59 /conf/log4j.properties
        -rw-r--r--   1 hdfsuser supergroup         10 2008-05-22 19:59 /conf/masters
        -rw-r--r--   1 hdfsuser supergroup         10 2008-05-22 19:59 /conf/slaves
        -rw-r--r--   1 hdfsuser supergroup        401 2008-05-22 19:59 /conf/sslinfo.xml.example
        
        Show
        Mukund Madhugiri added a comment - Updated patch based on feedback from Owen. Output looks like this: -rw-r--r-- 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf2 drwxr-xr-x - hdfsuser supergroup 0 2008-05-22 19:59 /conf/conf3 -rw-r--r-- 10 hdfsuser supergroup 535 2008-05-22 19:59 /conf/configuration.xsl -rw-r--r-- 1 hdfsuser supergroup 38550 2008-05-22 19:59 /conf/hadoop-default.xml -rw-r--r-- 1 hdfsuser supergroup 2237 2008-05-22 19:59 /conf/hadoop-env.sh -rw-r--r-- 1 hdfsuser supergroup 1245 2008-05-22 19:59 /conf/hadoop-metrics.properties -rw-r--r-- 1 hdfsuser supergroup 463 2008-05-22 19:59 /conf/hadoop-site.xml -rw-r--r-- 1 hdfsuser supergroup 2724 2008-05-22 19:59 /conf/log4j.properties -rw-r--r-- 1 hdfsuser supergroup 10 2008-05-22 19:59 /conf/masters -rw-r--r-- 1 hdfsuser supergroup 10 2008-05-22 19:59 /conf/slaves -rw-r--r-- 1 hdfsuser supergroup 401 2008-05-22 19:59 /conf/sslinfo.xml.example
        Hide
        Owen O'Malley added a comment -

        I just committed this. Thanks, Mukund!

        Show
        Owen O'Malley added a comment - I just committed this. Thanks, Mukund!

          People

          • Assignee:
            Mukund Madhugiri
            Reporter:
            Mukund Madhugiri
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development