Change dfs -ls listing to closely match format on Linux

• Key: HADOOP-3459
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Mukund Madhugiri
• Reporter: Mukund Madhugiri
• Affects Version/s: 0.18.0
• Fix Version/s: 0.18.0

Description

Change the dfs -ls command output to closely match the output format on Linux

Comments

[Mukund Madhugiri]

The current output format (in trunk) is:
Replication Size Date Permissions Owner Group Size Name

Example:
1 196 2008-05-22 19:59 rw-rr- hdfsuser supergroup /conf/commons-logging.properties

I have a patch that changes it to:

Permissions Replication Owner Group Size Date Name

Example:
rw-rr- 1 hdfsuser supergroup 196 2008-05-22 19:59 /conf/commons-logging.properties

 
CURRENT:
1    196         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/commons-logging.properties
     0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf
     0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf2
     0           2008-05-22 19:59  drwxr-xr-x  hdfsuser  supergroup  /conf/conf3
1    535         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/configuration.xsl
1    38550       2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-default.xml
1    2237        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-env.sh
1    1245        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-metrics.properties
1    463         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/hadoop-site.xml
1    2724        2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/log4j.properties
1    10          2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/masters
1    10          2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/slaves
1    401         2008-05-22 19:59  -rw-r--r--  hdfsuser  supergroup  /conf/sslinfo.xml.example

AFTER PATCH:
-rw-r--r--  1    hdfsuser  supergroup           196  2008-05-22 19:59  /conf/commons-logging.properties
drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf
drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf2
drwxr-xr-x  -    hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf3
-rw-r--r--  1    hdfsuser  supergroup           535  2008-05-22 19:59  /conf/configuration.xsl
-rw-r--r--  1    hdfsuser  supergroup         38550  2008-05-22 19:59  /conf/hadoop-default.xml
-rw-r--r--  1    hdfsuser  supergroup          2237  2008-05-22 19:59  /conf/hadoop-env.sh
-rw-r--r--  1    hdfsuser  supergroup          1245  2008-05-22 19:59  /conf/hadoop-metrics.properties
-rw-r--r--  1    hdfsuser  supergroup           463  2008-05-22 19:59  /conf/hadoop-site.xml
-rw-r--r--  1    hdfsuser  supergroup          2724  2008-05-22 19:59  /conf/log4j.properties
-rw-r--r--  1    hdfsuser  supergroup            10  2008-05-22 19:59  /conf/masters
-rw-r--r--  1    hdfsuser  supergroup            10  2008-05-22 19:59  /conf/slaves
-rw-r--r--  1    hdfsuser  supergroup           401  2008-05-22 19:59  /conf/sslinfo.xml.example

[Lohit Vijayarenu]

+1 this looks good

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12382951/HADOOP-3459.patch
against trunk revision 659405.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2519/console

This message is automatically generated.

[Mukund Madhugiri]

Regarding -1 for the tests, I am following up this JIRA was a complete set of tests for DFS commands via HADOOP-3100. The output of -ls and other commands are tested in that JIRA.

[Tsz Wo (Nicholas), SZE]

If some replication has two digits, the format won't look good since "-" is a fixed String.

[Mukund Madhugiri]

Nicholas,
Good point. I could right justify it, so it does not look bad. Any other suggestions welcome or I will go ahead and upload a new patch which right justifies the replication factor, along the lines of the file size

-rw-r--r--      1  hdfsuser  supergroup           196  2008-05-22 19:59  /conf/commons-logging.properties
drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf
drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf2
drwxr-xr-x      -  hdfsuser  supergroup             0  2008-05-22 19:59  /conf/conf3
-rw-r--r--     10  hdfsuser  supergroup           535  2008-05-22 19:59  /conf/configuration.xsl

[Tsz Wo (Nicholas), SZE]

The new output looks good.

[Owen O'Malley]

I really think it is required to have a fixed space between the fields. So I would much rather have spaces included in the format string than extra width.

System.out.printf("%-"+ maxReplication + "s ", 
          (!stat.isDir() ? stat.getReplication() : "-"));

instead of

System.out.printf("%-"+ (maxReplication + BORDER) +
         "s", (!stat.isDir() ? stat.getReplication() : "-"));

[Owen O'Malley]

I guess, I should have explained better. The problem that I've observed in the past, is that if the field is too wide, you lose the separator between the fields, making it impossible to figure out what is going wrong.

[Mukund Madhugiri]

Updated patch based on feedback from Owen. Output looks like this:

-rw-r--r--   1 hdfsuser supergroup        196 2008-05-22 19:59 /conf/commons-logging.properties
drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf
drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf2
drwxr-xr-x   - hdfsuser supergroup          0 2008-05-22 19:59 /conf/conf3
-rw-r--r--  10 hdfsuser supergroup        535 2008-05-22 19:59 /conf/configuration.xsl
-rw-r--r--   1 hdfsuser supergroup      38550 2008-05-22 19:59 /conf/hadoop-default.xml
-rw-r--r--   1 hdfsuser supergroup       2237 2008-05-22 19:59 /conf/hadoop-env.sh
-rw-r--r--   1 hdfsuser supergroup       1245 2008-05-22 19:59 /conf/hadoop-metrics.properties
-rw-r--r--   1 hdfsuser supergroup        463 2008-05-22 19:59 /conf/hadoop-site.xml
-rw-r--r--   1 hdfsuser supergroup       2724 2008-05-22 19:59 /conf/log4j.properties
-rw-r--r--   1 hdfsuser supergroup         10 2008-05-22 19:59 /conf/masters
-rw-r--r--   1 hdfsuser supergroup         10 2008-05-22 19:59 /conf/slaves
-rw-r--r--   1 hdfsuser supergroup        401 2008-05-22 19:59 /conf/sslinfo.xml.example

[Owen O'Malley]

I just committed this. Thanks, Mukund!