Support for non-recursive mkdir in HDFS

• Key: HDFS-618
• Type: Improvement
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Kan Zhang
• Reporter: Kan Zhang
• Component/s: hdfs client, name-node
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0

Description

Existing mkdirs call automatically creates missing parent directories. HADOOP-4952 call for a mkdir call that doesn't.

Comments

[Kan Zhang]

Attached a patch that
1. Added an interface method mkdir to DistributedFileSystem. It throws FileNotFoundException when parent dir doesn't exist.
2. Added a boolean param createParent to mkdirs() RPC call on ClientProtocol.

[Tsz Wo (Nicholas), SZE]

• new public methods needs javadoc, e.g.

  +  public boolean mkdir(Path f, FsPermission permission) throws IOException {
  +    return dfs.mkdirs(getPathName(f), permission, false);
  +  }

• The following method in DFSClient is not used. How about we deprecate it?

  public boolean mkdirs(String src) throws IOException {
  -    return mkdirs(src, null);
  +    return mkdirs(src, null, true);
     }

• This and HDFS-617 are going to conflict each other. It looks like that this patch is simpler and may go first. BTW, could you create helper methods for the common codes like the following?

  +    if (!createParent) {
  +      Path parent = new Path(src).getParent();
  +      if (parent != null && !dir.isDir(parent.toString())) {
  +        throw new FileNotFoundException("Can't mkdir " + src
  +            + "; parent directory " + parent.toString() + " doesn't exist.");
  +      }
  +    }

[Kan Zhang]

uploaded a new patch that adds the javadoc and deprecated mkdirs(String). Will upload a new patch after HDFS-617 is in, to resolve any conflict and also re-use the factored out verifyParentDir code.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419577/h618-03.patch
against trunk revision 814449.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/console

This message is automatically generated.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419591/h618-04.patch
against trunk revision 814449.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/console

This message is automatically generated.

[Kan Zhang]

attached a new patch for the latest trunk. also updated the test to check for FileAlreadyExistsException.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419697/h618-06.patch
against trunk revision 815496.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated 1 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/console

This message is automatically generated.

[Kan Zhang]

Uploaded a new patch that had one line documentation change to fix the javadoc warning. No code change. Here is the new test-patch result.

[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

[Tsz Wo (Nicholas), SZE]

+1 patch looks good.

[Tsz Wo (Nicholas), SZE]

I have committed this. Thanks, Kan!

[Hudson]

Integrated in Hadoop-Hdfs-trunk-Commit #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/37/)
. Support non-recursive mkdir(). Contributed by Kan Zhang

[Hudson]

Integrated in Hadoop-Hdfs-trunk #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/86/)
. Support non-recursive mkdir(). Contributed by Kan Zhang

[Hudson]

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #35 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/35/)

[Hudson]

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)

[Robert Chansler]

Editorial pass over all release notes prior to publication of 0.21.