Issue Details (XML | Word | Printable)

Key: HDFS-617
Type: Improvement Improvement
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Kan Zhang
Reporter: Kan Zhang
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Hadoop HDFS

Support for non-recursive create() in HDFS

Created: 14/Sep/09 05:35 PM   Updated: 09/Oct/09 10:56 PM
Return to search
Component/s: hdfs client, name-node
Affects Version/s: None
Fix Version/s: 0.21.0

Time Tracking:
Not Specified

File Attachments:
  Size
Text File Licensed for inclusion in ASF works h617-01.patch 2009-09-14 06:01 PM Kan Zhang 16 kB
Text File Licensed for inclusion in ASF works h617-02.patch 2009-09-14 11:32 PM Kan Zhang 16 kB
Text File Licensed for inclusion in ASF works h617-03.patch 2009-09-15 12:08 AM Kan Zhang 16 kB
Text File Licensed for inclusion in ASF works h617-04.patch 2009-09-15 12:37 AM Kan Zhang 16 kB
Text File Licensed for inclusion in ASF works h617-06.patch 2009-09-15 06:45 PM Kan Zhang 18 kB
Issue Links:
Reference
 

Hadoop Flags: Reviewed, Incompatible change
Release Note: New DFSClient.create(...) allows option of not creating missing parent(s).
Resolution Date: 15/Sep/09 09:23 PM


 Description  « Hide
HADOOP-4952 calls for a create call that doesn't automatically create missing parent directories.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Kan Zhang added a comment - 14/Sep/09 06:52 PM
Atteched a patch that
1. added a boolean parameter to the create() RPC call to indicate whether missing parent directories should be created.
2. added a createNonCursive() method to DistributedFileSystem interface, which doesn't create missing parent directories (complements existing create() method that does).

Hadoop QA added a comment - 14/Sep/09 09:01 PM
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419552/h617-01.patch
against trunk revision 814449.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/console

This message is automatically generated.


Tsz Wo (Nicholas), SZE added a comment - 14/Sep/09 11:21 PM
Patch looks good except that createParent should be false when for append.
startFileInternal(src, null, holder, clientMachine, EnumSet.of(CreateFlag.APPEND), 
-                      (short)blockManager.maxReplication, (long)0);
+                      true, (short)blockManager.maxReplication, (long)0);
     getEditLog().logSync();

Kan Zhang added a comment - 14/Sep/09 11:34 PM
Addressed Nicholas' comment with new patch.

Kan Zhang added a comment - 15/Sep/09 12:11 AM
attached a new patch that factors out the code for verifying parent dir exists.

Kan Zhang added a comment - 15/Sep/09 12:37 AM
one minor javadoc change.

Tsz Wo (Nicholas), SZE added a comment - 15/Sep/09 01:10 AM
If the parent is a file, a FileAlreadyExistsException should be thrown. See also HDFS-602.

Hadoop QA added a comment - 15/Sep/09 02:13 AM
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419582/h617-02.patch
against trunk revision 814449.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/console

This message is automatically generated.


Hadoop QA added a comment - 15/Sep/09 05:29 AM
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419593/h617-04.patch
against trunk revision 814449.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/console

This message is automatically generated.


Kan Zhang added a comment - 15/Sep/09 06:47 PM
attached a new patch that throws FileAlreadyExistsException when parent dir exists as a file.

Hadoop QA added a comment - 15/Sep/09 09:02 PM
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419664/h617-06.patch
against trunk revision 815001.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/console

This message is automatically generated.


Tsz Wo (Nicholas), SZE added a comment - 15/Sep/09 09:16 PM
+1 the patch is perfect!

Tsz Wo (Nicholas), SZE added a comment - 15/Sep/09 09:23 PM
I have committed this. Thanks, Kan!

Please add release note.


Hudson added a comment - 15/Sep/09 10:28 PM
Integrated in Hadoop-Hdfs-trunk-Commit #36 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/36/)
. Support non-recursive create(). Contributed by Kan Zhang

Hudson added a comment - 16/Sep/09 12:48 PM
Integrated in Hadoop-Hdfs-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/85/)
. Support non-recursive create(). Contributed by Kan Zhang

Hudson added a comment - 19/Sep/09 02:12 AM
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)

Robert Chansler added a comment - 09/Oct/09 10:56 PM
Editorial pass over all release notes prior to publication of 0.21.