[#HADOOP-2190] dfs ls and lsr commands differ from POSIX standards

Hadoop Common

dfs ls and lsr commands differ from POSIX standards

Created: 12/Nov/07 06:56 PM Updated: 08/Jul/09 04:42 PM

Component/s:

None

Affects Version/s:

0.15.1, 0.16.0

Fix Version/s:

0.17.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-2190-1.patch	2007-11-16 12:03 AM	Lohit Vijayarenu	4 kB
HADOOP-2190-2.patch	2007-11-18 07:31 AM	Lohit Vijayarenu	8 kB
Hadoop-2190_1.patch	2008-02-13 06:52 PM	Mahadev konar	2 kB
Hadoop-2190_2.patch	2008-02-13 07:15 PM	Mahadev konar	3 kB

Issue Links:

Cloners

This issue is cloned as:

~~HADOOP-2191~~ dfs du and dus commands differ from POSIX standards

Dependants

This issue blocks:

~~HADOOP-1677~~ improve semantics of the hadoop dfs command

Resolution Date:

13/Feb/08 09:14 PM

Description

« Hide

Assuming the dfs commands follow POSIX standards, there are some problems with the DFS ls and lsr commands. I compared the DFS output with that of RHEL 4u5

1. ls a directory when there are no files/directories in that directory:
Linux: No output
DFS: Found 0 items

2. ls a file/directory that does not exist:
Linux: ls: /doesnotexist: No such file or directory
DFS: Found 0 items

3. lsr a directory that does not exist:
Linux: ls: /doesnotexist: No such file or directory
DFS: No output

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Lohit Vijayarenu added a comment - 16/Nov/07 12:03 AM

Attached patch throws out error for ls, lsr, du, dus commands for non-existing files. There were 2 changes DistributedFileSyste.listPaths() was not returning null for non-existing files and FileSystem.listPaths() was consuming null returned from underlying listPaths and returning zero length Path[] array. So, I have made changes at few places so that listPaths returns null when required. I ran the tests and they look good. If anyone could take a look at the changes and approve it, I will prepare test cases and update another patch. The issue reported was also seen on local filesystem. fixing the second issue also takes care of local filesystem.

Below is the output after the patch

ls/lsr command 
[ hadoop-trunk]$ hadoop dfs -ls empty
Found 0 items
[ hadoop-trunk]$ hadoop dfs -ls nofile
ls: Could not get listing for nofile
[ hadoop-trunk]$ hadoop dfs -lsr nofile
lsr: Could not get listing for nofile

du command
[ hadoop-trunk]$ hadoop dfs -du empty
Found 0 items
[ hadoop-trunk]$ hadoop dfs -du nofile
du: Could not get listing for nofile

[ hadoop-trunk]$ hadoop dfs -dus empty
empty   0
[ hadoop-trunk]$ hadoop dfs -dus nofile
dus: dus: No match: nofile
[ hadoop-trunk]$

[ Permlink | « Hide ]

Mukund Madhugiri added a comment - 16/Nov/07 01:45 AM

Can the command output be along the lines of what we see on linux?

RHEL throws this:
ls: /doesnotexist: No such file or directory

[ Permlink | « Hide ]

Lohit Vijayarenu added a comment - 16/Nov/07 07:11 AM

sure, if patch is approved I will make these changes while adding testcase.

[ Permlink | « Hide ]

dhruba borthakur added a comment - 16/Nov/07 10:09 PM

It will be a good idea to remove the dependency of this patch on the deprecated FileSystem.listPath API

[ Permlink | « Hide ]

Lohit Vijayarenu added a comment - 18/Nov/07 07:31 AM

Another patch, which now calls listStatus instead of deprecated listPaths(Path) . updated test case and changes suggested by Mukund.

[ Permlink | « Hide ]

Mahadev konar added a comment - 13/Feb/08 06:52 PM

this patch fixes the issue but only on the FsShell side.The previous patch fix it on the filesystem side which would requrioe changes to all the filesysstems. Also, this just checks to see if the directory/ does not exist only if it gets 0 listings from listStatus.