Issue Details (XML | Word | Printable)

Key: HADOOP-2865
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Edward J. Yoon
Reporter: Konstantin Shvachko
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

FsShell.ls() should print file attributes first then the path name.

Created: 21/Feb/08 02:09 AM   Updated: 22/Aug/08 07:50 PM
Return to search
Component/s: None
Affects Version/s: 0.15.0
Fix Version/s: 0.18.0

Time Tracking:
Not Specified

File Attachments:
  Size
Text File Licensed for inclusion in ASF works 2865_v01.patch 2008-04-10 11:02 AM Edward J. Yoon 2 kB
Text File Licensed for inclusion in ASF works 2865_v02.patch 2008-04-15 04:19 AM Edward J. Yoon 4 kB
Text File Licensed for inclusion in ASF works 2865_v03.patch 2008-04-15 04:37 AM Edward J. Yoon 4 kB
Text File Licensed for inclusion in ASF works 2865_v04.patch 2008-04-16 12:40 AM Edward J. Yoon 3 kB

Hadoop Flags: Reviewed, Incompatible change
Release Note:
Changed the output of the "fs -ls" command to more closely match familiar Linux format. Additional changes were made by HADOOP-3459. Applications that parse the command output should be reviewed.
Resolution Date: 16/Apr/08 10:31 PM


 Description  « Hide
When we had a handful of attributes this looked OK. But after the permissions the ls output is just unreadable.
This how the output looks now:
/CHANGES.txt    <r 2>   174779  2008-01-29 10:00        rw-r--r--       shv     supergroup
/LICENSE.txt    <r 2>   11358   2008-01-28 17:27        rw-r--r--       shv     supergroup
/NOTICE.txt     <r 2>   101     2008-01-29 12:06        rw-r--r--       shv     supergroup
/Work   <dir>           2008-01-23 17:43        rwxr-xr-x       shv     supergroup
/Work/hadoop-data       <dir>           2008-01-23 17:43        rwxr-xr-x       shv     supergroup

This is how it should look:

2   174779  2008-01-29 10:00       -rw-r--r--       shv     supergroup     /CHANGES.txt
2   11358   2008-01-28 17:27       -rw-r--r--       shv     supergroup     /LICENSE.txt
2   101     2008-01-29 12:06       -rw-r--r--       shv     supergroup     /NOTICE.txt
            2008-01-23 17:43       drwxr-xr-x       shv     supergroup     /Work
            2008-01-23 17:43       drwxr-xr-x       shv     supergroup     /Work/hadoop-data


 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Koji Noguchi added a comment - 25/Feb/08 08:17 PM
I'm not against this, but just a note that some users have scripts written that depend on the order of the '-ls' result.
Some even failed when we just added a timestamp...

Tsz Wo (Nicholas), SZE added a comment - 26/Feb/08 09:46 PM
  • Similar problems exist on du, e.g.
    bash-3.2$ ./bin/hadoop fs -du /    
    Found 3 items
    hdfs://host:9000/a.txt  159
    hdfs://host:9000/build.xml      44198
    hdfs://host:9000/user   318
  • The column for the file size should be wider (at least 15 digits, probably 18 digits) and right aligned.

Edward J. Yoon added a comment - 10/Apr/08 11:02 AM
[udanax@a51066 hadoop]# bin/hadoop dfs -du /
Found 4 items
11358             hdfs://localhost:9000/LICENSE.txt
1366              hdfs://localhost:9000/README.txt
0                 hdfs://localhost:9000/home
0                 hdfs://localhost:9000/user
[udanax@a51066 hadoop]# bin/hadoop dfs -ls
Found 5 items
1       11358   2008-04-10 18:27        -rw-r--r--      root    supergroup      /user/root/LICENSE.txt
1       101     2008-04-10 18:27        -rw-r--r--      root    supergroup      /user/root/NOTICE.txt
1       1366    2008-04-10 18:26        -rw-r--r--      root    supergroup      /user/root/README.txt
1       831954  2008-04-10 18:41        -rw-r--r--      root    supergroup      /user/root/hadoop-0.18.0-dev-test.jar
        0       2008-04-10 18:26        drwxr-xr-x      root    supergroup      /user/root/logs
[udanax@a51066 hadoop]#

Edward J. Yoon added a comment - 10/Apr/08 11:06 AM
Doesn't it need a test case?
If so, i'll submit this patch.

Edward J. Yoon added a comment - 10/Apr/08 11:06 AM
Submitting.

Konstantin Shvachko added a comment - 14/Apr/08 10:43 PM
+0.7
I think this is good as a first step in improving the formatting of ls.
There are 3 cases when formatting brakes
  1. when file size is larger than 7 digits
  2. when user names substantially differ in length
  3. when group names differ
2	168	2007-10-09 20:21	-rw-rw-rw-	shv	supergroup	/README.txt
	0	2008-04-09 18:47	drwxr-xr-x	administrator	supergroup	/conf
3	12345678	2008-04-09 18:46	-rw-r--r--	shv	supergroup	/conf/.svn/README.txt
3	1234567	2008-04-09 18:47	-rw-r--r--	shv	supergroup	/conf/.svn/dir-prop-base
3	90	2008-04-09 18:46	-rw-r--r--	shv	user	/conf/.svn/dir-wcprops
3	9012345	2008-04-09 18:47	-rw-r--r--	shv	supergroup	/conf/.svn/dir-props

The first one is going to be very common, since we expect large files and design for it. And we should fix it.
The best way would be to convert bytes into MB, GB, TB, or even PB if needed, but at least using "%-XXd" as in DU would be better.
This should also produce right aligned column.

Dynamically calculating column widths is probably the next logical improvement step, which belongs to a different issue.

We should mark this issue as an incomatible change.


Edward J. Yoon added a comment - 15/Apr/08 04:19 AM
This v02.patch include the AutoFit (by using in-memory) as describe below.
[root@udanax hadoop]# bin/hadoop dfs -du
Found 2 items
11358  hdfs://localhost:9000/user/root/LICENSE.txt
1366   hdfs://localhost:9000/user/root/udanax
[root@udanax hadoop]# bin/hadoop dfs -ls
Found 2 items
1  11358  2008-04-15 10:53  -rw-r--r--  udanax  supergroup  /user/root/LICENSE.txt
   0      2008-04-15 12:54  drwxr-xr-x  root    supergroup  /user/root/udanax
[root@udanax hadoop]#

BTW, what does Incompatible change" mean? and What should i do?


Edward J. Yoon added a comment - 15/Apr/08 04:37 AM
when a lot of files are exist, v02.patch seems problematic.

Edward J. Yoon added a comment - 15/Apr/08 09:06 AM
Checking incompatible change.

Konstantin Shvachko added a comment - 15/Apr/08 11:03 PM
In your code could you please set initial values to
int maxReplication = 3, maxLen = 10, maxOwner = 0,maxGroup = 0;

This should work for most files.
You should also resubmit the patch.


Edward J. Yoon added a comment - 16/Apr/08 12:40 AM
Thank you for your review, Fixed them.

Edward J. Yoon added a comment - 16/Apr/08 12:42 AM
Resubmitting.

Hadoop QA added a comment - 16/Apr/08 04:08 AM
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380230/2865_v04.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2246/console

This message is automatically generated.


Konstantin Shvachko added a comment - 16/Apr/08 10:31 PM
I just committed this. Thank you Edward.

Hudson added a comment - 17/Apr/08 12:11 PM