Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-618

Support for non-recursive mkdir in HDFS

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New DFSClient.mkdir(...) allows option of not creating missing parent(s).

      Description

      Existing mkdirs call automatically creates missing parent directories. HADOOP-4952 call for a mkdir call that doesn't.

      1. h618-03.patch
        10 kB
        Kan Zhang
      2. h618-04.patch
        10 kB
        Kan Zhang
      3. h618-06.patch
        11 kB
        Kan Zhang
      4. h618-07.patch
        11 kB
        Kan Zhang

        Issue Links

          Activity

          Hide
          Kan Zhang added a comment -

          Attached a patch that
          1. Added an interface method mkdir to DistributedFileSystem. It throws FileNotFoundException when parent dir doesn't exist.
          2. Added a boolean param createParent to mkdirs() RPC call on ClientProtocol.

          Show
          Kan Zhang added a comment - Attached a patch that 1. Added an interface method mkdir to DistributedFileSystem. It throws FileNotFoundException when parent dir doesn't exist. 2. Added a boolean param createParent to mkdirs() RPC call on ClientProtocol.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • new public methods needs javadoc, e.g.
            +  public boolean mkdir(Path f, FsPermission permission) throws IOException {
            +    return dfs.mkdirs(getPathName(f), permission, false);
            +  }
            
          • The following method in DFSClient is not used. How about we deprecate it?
               public boolean mkdirs(String src) throws IOException {
            -    return mkdirs(src, null);
            +    return mkdirs(src, null, true);
               }
            
          • This and HDFS-617 are going to conflict each other. It looks like that this patch is simpler and may go first. BTW, could you create helper methods for the common codes like the following?
            +    if (!createParent) {
            +      Path parent = new Path(src).getParent();
            +      if (parent != null && !dir.isDir(parent.toString())) {
            +        throw new FileNotFoundException("Can't mkdir " + src
            +            + "; parent directory " + parent.toString() + " doesn't exist.");
            +      }
            +    }
            
          Show
          Tsz Wo Nicholas Sze added a comment - new public methods needs javadoc, e.g. + public boolean mkdir(Path f, FsPermission permission) throws IOException { + return dfs.mkdirs(getPathName(f), permission, false ); + } The following method in DFSClient is not used. How about we deprecate it? public boolean mkdirs( String src) throws IOException { - return mkdirs(src, null ); + return mkdirs(src, null , true ); } This and HDFS-617 are going to conflict each other. It looks like that this patch is simpler and may go first. BTW, could you create helper methods for the common codes like the following? + if (!createParent) { + Path parent = new Path(src).getParent(); + if (parent != null && !dir.isDir(parent.toString())) { + throw new FileNotFoundException( "Can't mkdir " + src + + "; parent directory " + parent.toString() + " doesn't exist." ); + } + }
          Hide
          Kan Zhang added a comment -

          uploaded a new patch that adds the javadoc and deprecated mkdirs(String). Will upload a new patch after HDFS-617 is in, to resolve any conflict and also re-use the factored out verifyParentDir code.

          Show
          Kan Zhang added a comment - uploaded a new patch that adds the javadoc and deprecated mkdirs(String). Will upload a new patch after HDFS-617 is in, to resolve any conflict and also re-use the factored out verifyParentDir code.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419577/h618-03.patch
          against trunk revision 814449.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419577/h618-03.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419591/h618-04.patch
          against trunk revision 814449.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419591/h618-04.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          attached a new patch for the latest trunk. also updated the test to check for FileAlreadyExistsException.

          Show
          Kan Zhang added a comment - attached a new patch for the latest trunk. also updated the test to check for FileAlreadyExistsException.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419697/h618-06.patch
          against trunk revision 815496.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419697/h618-06.patch against trunk revision 815496. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/8/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          Uploaded a new patch that had one line documentation change to fix the javadoc warning. No code change. Here is the new test-patch result.

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 6 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Kan Zhang added a comment - Uploaded a new patch that had one line documentation change to fix the javadoc warning. No code change. Here is the new test-patch result. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Kan!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Kan!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/37/)
          . Support non-recursive mkdir(). Contributed by Kan Zhang

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/37/ ) . Support non-recursive mkdir(). Contributed by Kan Zhang
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/86/)
          . Support non-recursive mkdir(). Contributed by Kan Zhang

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #86 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/86/ ) . Support non-recursive mkdir(). Contributed by Kan Zhang
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #35 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/35/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #35 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/35/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/ )
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development