Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-617

Support for non-recursive create() in HDFS

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.21.0, 1.0.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New DFSClient.create(...) allows option of not creating missing parent(s).

      Description

      HADOOP-4952 calls for a create call that doesn't automatically create missing parent directories.

      1. HDFS-617-branch-0.20-security.patch
        22 kB
        Jitendra Nath Pandey
      2. HDFS-617_20-append.patch
        20 kB
        Nicolas Spiegelberg
      3. h617-06.patch
        18 kB
        Kan Zhang
      4. h617-04.patch
        16 kB
        Kan Zhang
      5. h617-03.patch
        16 kB
        Kan Zhang
      6. h617-02.patch
        16 kB
        Kan Zhang
      7. h617-01.patch
        16 kB
        Kan Zhang

        Issue Links

          Activity

          Hide
          Nicolas Spiegelberg added a comment -

          @Jitendra & @Suresh : since you now have the 3 critical jiras for HBASE-2312 committed to 20-security, it would be good to remove @Ignore from testLogRollAfterSplitStart in HBase to verify that these 3 patches fix HBase lease recovery with code as-is.

          Show
          Nicolas Spiegelberg added a comment - @Jitendra & @Suresh : since you now have the 3 critical jiras for HBASE-2312 committed to 20-security, it would be good to remove @Ignore from testLogRollAfterSplitStart in HBase to verify that these 3 patches fix HBase lease recovery with code as-is.
          Hide
          Suresh Srinivas added a comment -

          +1 for 20-security patch.

          Show
          Suresh Srinivas added a comment - +1 for 20-security patch.
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for 20-security uploaded.
          The deprecation of older create API is removed. I didn't bump the version of client protocol for backward compatibility with 205. I also added FileAlreadyExistsException to Hdfs, because otherwise hdfs had a dependency on mapred.

          Show
          Jitendra Nath Pandey added a comment - Patch for 20-security uploaded. The deprecation of older create API is removed. I didn't bump the version of client protocol for backward compatibility with 205. I also added FileAlreadyExistsException to Hdfs, because otherwise hdfs had a dependency on mapred.
          Hide
          Hairong Kuang added a comment -

          The problem is that the new client does not work with the old server...I think in the Hadoop world, we already bump up the version number when there is any change to the protocol. I do not see why the append branch needs to be compatible with 0.20.3. We have already broken the compatibility with HDFS-724 etc.

          Show
          Hairong Kuang added a comment - The problem is that the new client does not work with the old server...I think in the Hadoop world, we already bump up the version number when there is any change to the protocol. I do not see why the append branch needs to be compatible with 0.20.3. We have already broken the compatibility with HDFS-724 etc.
          Hide
          Nicolas Spiegelberg added a comment -

          @Hairong: Dhruba asked that I not bump the version number so 0.20-append DNs will be backwards compatible with 0.20.3. If you notice, I keep the old APIs around. I also revert back to the old recursive create on the HBase side if anything the new api usage fails.

          @Todd: we could probably remove the deprecation warnings. Technically, the deprecation rev is 0.21, because these apis are removed in 0.22.

          Show
          Nicolas Spiegelberg added a comment - @Hairong: Dhruba asked that I not bump the version number so 0.20-append DNs will be backwards compatible with 0.20.3. If you notice, I keep the old APIs around. I also revert back to the old recursive create on the HBase side if anything the new api usage fails. @Todd: we could probably remove the deprecation warnings. Technically, the deprecation rev is 0.21, because these apis are removed in 0.22.
          Hide
          Hairong Kuang added a comment -

          Hi Nicolas, for the 0.20 append patch, do you also need to update ClientProtocol?

          Show
          Hairong Kuang added a comment - Hi Nicolas, for the 0.20 append patch, do you also need to update ClientProtocol?
          Hide
          Todd Lipcon added a comment -

          Patch for 0.20 append looks good, except I don't think we should mark the old APIs as @Deprecated. We'd like to ship 0.20-append as a drop-in replacement for 0.20 for people using HBase, and adding deprecation warnings to 0.20 APIs seems over-zealous.

          Show
          Todd Lipcon added a comment - Patch for 0.20 append looks good, except I don't think we should mark the old APIs as @Deprecated. We'd like to ship 0.20-append as a drop-in replacement for 0.20 for people using HBase, and adding deprecation warnings to 0.20 APIs seems over-zealous.
          Hide
          Nicolas Spiegelberg added a comment -

          Backport of patch for 0.20-append branch

          Show
          Nicolas Spiegelberg added a comment - Backport of patch for 0.20-append branch
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/85/)
          . Support non-recursive create(). Contributed by Kan Zhang

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/85/ ) . Support non-recursive create(). Contributed by Kan Zhang
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #36 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/36/)
          . Support non-recursive create(). Contributed by Kan Zhang

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #36 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/36/ ) . Support non-recursive create(). Contributed by Kan Zhang
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Kan!

          Please add release note.

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Kan! Please add release note.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the patch is perfect!

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the patch is perfect!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419664/h617-06.patch
          against trunk revision 815001.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419664/h617-06.patch against trunk revision 815001. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          attached a new patch that throws FileAlreadyExistsException when parent dir exists as a file.

          Show
          Kan Zhang added a comment - attached a new patch that throws FileAlreadyExistsException when parent dir exists as a file.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419593/h617-04.patch
          against trunk revision 814449.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419593/h617-04.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419582/h617-02.patch
          against trunk revision 814449.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419582/h617-02.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          If the parent is a file, a FileAlreadyExistsException should be thrown. See also HDFS-602.

          Show
          Tsz Wo Nicholas Sze added a comment - If the parent is a file, a FileAlreadyExistsException should be thrown. See also HDFS-602 .
          Hide
          Kan Zhang added a comment -

          one minor javadoc change.

          Show
          Kan Zhang added a comment - one minor javadoc change.
          Hide
          Kan Zhang added a comment -

          attached a new patch that factors out the code for verifying parent dir exists.

          Show
          Kan Zhang added a comment - attached a new patch that factors out the code for verifying parent dir exists.
          Hide
          Kan Zhang added a comment -

          Addressed Nicholas' comment with new patch.

          Show
          Kan Zhang added a comment - Addressed Nicholas' comment with new patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Patch looks good except that createParent should be false when for append.

               startFileInternal(src, null, holder, clientMachine, EnumSet.of(CreateFlag.APPEND), 
          -                      (short)blockManager.maxReplication, (long)0);
          +                      true, (short)blockManager.maxReplication, (long)0);
               getEditLog().logSync();
          
          Show
          Tsz Wo Nicholas Sze added a comment - Patch looks good except that createParent should be false when for append. startFileInternal(src, null , holder, clientMachine, EnumSet.of(CreateFlag.APPEND), - ( short )blockManager.maxReplication, ( long )0); + true , ( short )blockManager.maxReplication, ( long )0); getEditLog().logSync();
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419552/h617-01.patch
          against trunk revision 814449.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419552/h617-01.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          Atteched a patch that
          1. added a boolean parameter to the create() RPC call to indicate whether missing parent directories should be created.
          2. added a createNonCursive() method to DistributedFileSystem interface, which doesn't create missing parent directories (complements existing create() method that does).

          Show
          Kan Zhang added a comment - Atteched a patch that 1. added a boolean parameter to the create() RPC call to indicate whether missing parent directories should be created. 2. added a createNonCursive() method to DistributedFileSystem interface, which doesn't create missing parent directories (complements existing create() method that does).

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development