dfs mkdir command differs from POSIX standards

• Key: HADOOP-2195
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Mahadev konar
• Reporter: Mukund Madhugiri
• Affects Version/s: 0.15.1, 0.16.0
• Fix Version/s: 0.17.0

Description

Assuming the dfs commands follow POSIX standards, there are some problems with the DFS mkdir command. I compared the DFS output with that of RHEL 4u5:

1. mkdir a directory that exists:
Linux: mkdir: cannot create directory `test': File exists
DFS: No output

2. mkdir a directory with the name of a file that already exists:

• Linux: mkdir: `test/one' exists but is not a directory
• DFS: mkdir: java.io.IOException: Invalid directory name: <home directory>/test/one

Comments

[Chris Douglas]

I think the semantics for mkdir follow mkdir -p, so the first case is consistent with POSIX.

The second case- throwing an exception- seems reasonable to me, though the error message is confusing.

[Mukund Madhugiri]

for the first case, it needs to be documented that it follows mkdir -p. The du command documents it when help for the du command is displayed.

mkdir currently does not display what semantics it follows:
bin/hadoop dfs -help mkdir
-mkdir <path>: Create a directory in specified location.

whereas du shows this:
bin/hadoop dfs -help du
-du <path>: Show the amount of space, in bytes, used by the files that
match the specified file pattern. Equivalent to the unix
command "du -sb <path>/*" in case of a directory,
and to "du -b <path>" in case of a file.
The output is in the form
name(full path) size (in bytes)

[Nigel Daley]

The second case- throwing an exception- seems reasonable to me, though the error message is confusing.

I do not think that throwing an exception (or exposing exception wording) is reasonable. The user is a person, not another program.

[Mahadev konar]

this patch fixes the issues described inthe bug. mkdir on existing dir and file works the same as linux. This patch calls getFIleStatus before calling a mkdir and then checks to see if an exception is thrown, in that case proceeds with mkdir else errors out with the corresponding error messages.

[Lohit Vijayarenu]

+1 checking before invoking mkdir sounds reasonable.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12375535/Hadoop-2195_1.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 6 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1796/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1796/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1796/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1796/console

This message is automatically generated.

[Raghu Angadi]

I just committed. Thanks Mahadev.

[Hudson]

Integrated in Hadoop-trunk #401 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/401/)