Hadoop Common
  1. Hadoop Common
  2. HADOOP-2865

FsShell.ls() should print file attributes first then the path name.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459. Applications that parse the command output should be reviewed.
      Show
      Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459 . Applications that parse the command output should be reviewed.

      Description

      When we had a handful of attributes this looked OK. But after the permissions the ls output is just unreadable.
      This how the output looks now:

      /CHANGES.txt    <r 2>   174779  2008-01-29 10:00        rw-r--r--       shv     supergroup
      /LICENSE.txt    <r 2>   11358   2008-01-28 17:27        rw-r--r--       shv     supergroup
      /NOTICE.txt     <r 2>   101     2008-01-29 12:06        rw-r--r--       shv     supergroup
      /Work   <dir>           2008-01-23 17:43        rwxr-xr-x       shv     supergroup
      /Work/hadoop-data       <dir>           2008-01-23 17:43        rwxr-xr-x       shv     supergroup
      

      This is how it should look:

      2   174779  2008-01-29 10:00       -rw-r--r--       shv     supergroup     /CHANGES.txt
      2   11358   2008-01-28 17:27       -rw-r--r--       shv     supergroup     /LICENSE.txt
      2   101     2008-01-29 12:06       -rw-r--r--       shv     supergroup     /NOTICE.txt
                  2008-01-23 17:43       drwxr-xr-x       shv     supergroup     /Work
                  2008-01-23 17:43       drwxr-xr-x       shv     supergroup     /Work/hadoop-data
      
      1. 2865_v01.patch
        2 kB
        Edward J. Yoon
      2. 2865_v02.patch
        4 kB
        Edward J. Yoon
      3. 2865_v03.patch
        4 kB
        Edward J. Yoon
      4. 2865_v04.patch
        3 kB
        Edward J. Yoon

        Activity

        Konstantin Shvachko created issue -
        Hide
        Koji Noguchi added a comment -

        I'm not against this, but just a note that some users have scripts written that depend on the order of the '-ls' result.
        Some even failed when we just added a timestamp...

        Show
        Koji Noguchi added a comment - I'm not against this, but just a note that some users have scripts written that depend on the order of the '-ls' result. Some even failed when we just added a timestamp...
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • Similar problems exist on du, e.g.
          bash-3.2$ ./bin/hadoop fs -du /    
          Found 3 items
          hdfs://host:9000/a.txt  159
          hdfs://host:9000/build.xml      44198
          hdfs://host:9000/user   318
          
        • The column for the file size should be wider (at least 15 digits, probably 18 digits) and right aligned.
        Show
        Tsz Wo Nicholas Sze added a comment - Similar problems exist on du, e.g. bash-3.2$ ./bin/hadoop fs -du / Found 3 items hdfs: //host:9000/a.txt 159 hdfs: //host:9000/build.xml 44198 hdfs: //host:9000/user 318 The column for the file size should be wider (at least 15 digits, probably 18 digits) and right aligned.
        Robert Chansler made changes -
        Field Original Value New Value
        Fix Version/s 0.17.0 [ 12312913 ]
        Edward J. Yoon made changes -
        Assignee Edward J. Yoon [ udanax ]
        Hide
        Edward J. Yoon added a comment -
        [udanax@a51066 hadoop]# bin/hadoop dfs -du /
        Found 4 items
        11358             hdfs://localhost:9000/LICENSE.txt
        1366              hdfs://localhost:9000/README.txt
        0                 hdfs://localhost:9000/home
        0                 hdfs://localhost:9000/user
        [udanax@a51066 hadoop]# bin/hadoop dfs -ls
        Found 5 items
        1       11358   2008-04-10 18:27        -rw-r--r--      root    supergroup      /user/root/LICENSE.txt
        1       101     2008-04-10 18:27        -rw-r--r--      root    supergroup      /user/root/NOTICE.txt
        1       1366    2008-04-10 18:26        -rw-r--r--      root    supergroup      /user/root/README.txt
        1       831954  2008-04-10 18:41        -rw-r--r--      root    supergroup      /user/root/hadoop-0.18.0-dev-test.jar
                0       2008-04-10 18:26        drwxr-xr-x      root    supergroup      /user/root/logs
        [udanax@a51066 hadoop]# 
        
        Show
        Edward J. Yoon added a comment - [udanax@a51066 hadoop]# bin/hadoop dfs -du / Found 4 items 11358 hdfs: //localhost:9000/LICENSE.txt 1366 hdfs: //localhost:9000/README.txt 0 hdfs: //localhost:9000/home 0 hdfs: //localhost:9000/user [udanax@a51066 hadoop]# bin/hadoop dfs -ls Found 5 items 1 11358 2008-04-10 18:27 -rw-r--r-- root supergroup /user/root/LICENSE.txt 1 101 2008-04-10 18:27 -rw-r--r-- root supergroup /user/root/NOTICE.txt 1 1366 2008-04-10 18:26 -rw-r--r-- root supergroup /user/root/README.txt 1 831954 2008-04-10 18:41 -rw-r--r-- root supergroup /user/root/hadoop-0.18.0-dev-test.jar 0 2008-04-10 18:26 drwxr-xr-x root supergroup /user/root/logs [udanax@a51066 hadoop]#
        Edward J. Yoon made changes -
        Attachment 2865_v01.patch [ 12379823 ]
        Hide
        Edward J. Yoon added a comment -

        Doesn't it need a test case?
        If so, i'll submit this patch.

        Show
        Edward J. Yoon added a comment - Doesn't it need a test case? If so, i'll submit this patch.
        Hide
        Edward J. Yoon added a comment -

        Submitting.

        Show
        Edward J. Yoon added a comment - Submitting.
        Edward J. Yoon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Edward J. Yoon made changes -
        Fix Version/s 0.18.0 [ 12312972 ]
        Hide
        Konstantin Shvachko added a comment -

        +0.7
        I think this is good as a first step in improving the formatting of ls.
        There are 3 cases when formatting brakes

        1. when file size is larger than 7 digits
        2. when user names substantially differ in length
        3. when group names differ
        2	168	2007-10-09 20:21	-rw-rw-rw-	shv	supergroup	/README.txt
        	0	2008-04-09 18:47	drwxr-xr-x	administrator	supergroup	/conf
        3	12345678	2008-04-09 18:46	-rw-r--r--	shv	supergroup	/conf/.svn/README.txt
        3	1234567	2008-04-09 18:47	-rw-r--r--	shv	supergroup	/conf/.svn/dir-prop-base
        3	90	2008-04-09 18:46	-rw-r--r--	shv	user	/conf/.svn/dir-wcprops
        3	9012345	2008-04-09 18:47	-rw-r--r--	shv	supergroup	/conf/.svn/dir-props
        

        The first one is going to be very common, since we expect large files and design for it. And we should fix it.
        The best way would be to convert bytes into MB, GB, TB, or even PB if needed, but at least using "%-XXd" as in DU would be better.
        This should also produce right aligned column.

        Dynamically calculating column widths is probably the next logical improvement step, which belongs to a different issue.

        We should mark this issue as an incomatible change.

        Show
        Konstantin Shvachko added a comment - +0.7 I think this is good as a first step in improving the formatting of ls. There are 3 cases when formatting brakes when file size is larger than 7 digits when user names substantially differ in length when group names differ 2 168 2007-10-09 20:21 -rw-rw-rw- shv supergroup /README.txt 0 2008-04-09 18:47 drwxr-xr-x administrator supergroup /conf 3 12345678 2008-04-09 18:46 -rw-r--r-- shv supergroup /conf/.svn/README.txt 3 1234567 2008-04-09 18:47 -rw-r--r-- shv supergroup /conf/.svn/dir-prop-base 3 90 2008-04-09 18:46 -rw-r--r-- shv user /conf/.svn/dir-wcprops 3 9012345 2008-04-09 18:47 -rw-r--r-- shv supergroup /conf/.svn/dir-props The first one is going to be very common, since we expect large files and design for it. And we should fix it. The best way would be to convert bytes into MB, GB, TB, or even PB if needed, but at least using "%-XXd" as in DU would be better. This should also produce right aligned column. Dynamically calculating column widths is probably the next logical improvement step, which belongs to a different issue. We should mark this issue as an incomatible change.
        Hide
        Edward J. Yoon added a comment -

        This v02.patch include the AutoFit (by using in-memory) as describe below.

        [root@udanax hadoop]# bin/hadoop dfs -du
        Found 2 items
        11358  hdfs://localhost:9000/user/root/LICENSE.txt
        1366   hdfs://localhost:9000/user/root/udanax
        [root@udanax hadoop]# bin/hadoop dfs -ls
        Found 2 items
        1  11358  2008-04-15 10:53  -rw-r--r--  udanax  supergroup  /user/root/LICENSE.txt
           0      2008-04-15 12:54  drwxr-xr-x  root    supergroup  /user/root/udanax
        [root@udanax hadoop]# 
        

        BTW, what does Incompatible change" mean? and What should i do?

        Show
        Edward J. Yoon added a comment - This v02.patch include the AutoFit (by using in-memory) as describe below. [root@udanax hadoop]# bin/hadoop dfs -du Found 2 items 11358 hdfs: //localhost:9000/user/root/LICENSE.txt 1366 hdfs: //localhost:9000/user/root/udanax [root@udanax hadoop]# bin/hadoop dfs -ls Found 2 items 1 11358 2008-04-15 10:53 -rw-r--r-- udanax supergroup /user/root/LICENSE.txt 0 2008-04-15 12:54 drwxr-xr-x root supergroup /user/root/udanax [root@udanax hadoop]# BTW, what does Incompatible change" mean? and What should i do?
        Edward J. Yoon made changes -
        Attachment 2865_v02.patch [ 12380138 ]
        Hide
        Edward J. Yoon added a comment -

        when a lot of files are exist, v02.patch seems problematic.

        Show
        Edward J. Yoon added a comment - when a lot of files are exist, v02.patch seems problematic.
        Edward J. Yoon made changes -
        Attachment 2865_v03.patch [ 12380140 ]
        Hide
        Edward J. Yoon added a comment -

        Checking incompatible change.

        Show
        Edward J. Yoon added a comment - Checking incompatible change.
        Edward J. Yoon made changes -
        Hadoop Flags [Incompatible change]
        Hide
        Konstantin Shvachko added a comment -

        In your code could you please set initial values to

        int maxReplication = 3, maxLen = 10, maxOwner = 0,maxGroup = 0;
        

        This should work for most files.
        You should also resubmit the patch.

        Show
        Konstantin Shvachko added a comment - In your code could you please set initial values to int maxReplication = 3, maxLen = 10, maxOwner = 0,maxGroup = 0; This should work for most files. You should also resubmit the patch.
        Hide
        Edward J. Yoon added a comment -

        Thank you for your review, Fixed them.

        Show
        Edward J. Yoon added a comment - Thank you for your review, Fixed them.
        Edward J. Yoon made changes -
        Attachment 2865_v04.patch [ 12380230 ]
        Edward J. Yoon made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Edward J. Yoon added a comment -

        Resubmitting.

        Show
        Edward J. Yoon added a comment - Resubmitting.
        Edward J. Yoon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12380230/2865_v04.patch
        against trunk revision 645773.

        @author +1. The patch does not contain any @author tags.

        tests included -1. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380230/2865_v04.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included -1. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/console This message is automatically generated.
        Hide
        Konstantin Shvachko added a comment -

        I just committed this. Thank you Edward.

        Show
        Konstantin Shvachko added a comment - I just committed this. Thank you Edward.
        Konstantin Shvachko made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #463 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/463/ )
        Robert Chansler made changes -
        Release Note Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459. Applications that parse the command output should be reviewed.
        Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Edward J. Yoon
            Reporter:
            Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development