Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.20-append
-
Component/s: hdfs-client, namenode
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Incompatible change, Reviewed
-
Release Note:New DFSClient.create(...) allows option of not creating missing parent(s).
Description
HADOOP-4952 calls for a create call that doesn't automatically create missing parent directories.
Issue Links
- is depended upon by
-
HDFS-1269
Support non-recursive create() in FileSystem API
-
- Resolved
-
-
HADOOP-6886
LocalFileSystem Needs createNonRecursive API
-
- Closed
-
- is related to
-
HADOOP-6840
Support non-recursive create() in FileSystem & SequenceFile.Writer
-
- Closed
-
- is required by
-
HBASE-2312
Possible data loss when RS goes into GC pause while rolling HLog
-
- Closed
-
- relates to
-
HDFS-602
Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException
-
- Closed
-
-
HADOOP-4952
Improved files system interface for the application writer.
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
+1 for 20-security patch.
Patch for 20-security uploaded.
The deprecation of older create API is removed. I didn't bump the version of client protocol for backward compatibility with 205. I also added FileAlreadyExistsException to Hdfs, because otherwise hdfs had a dependency on mapred.
The problem is that the new client does not work with the old server...I think in the Hadoop world, we already bump up the version number when there is any change to the protocol. I do not see why the append branch needs to be compatible with 0.20.3. We have already broken the compatibility with HDFS-724 etc.
@Hairong: Dhruba asked that I not bump the version number so 0.20-append DNs will be backwards compatible with 0.20.3. If you notice, I keep the old APIs around. I also revert back to the old recursive create on the HBase side if anything the new api usage fails.
@Todd: we could probably remove the deprecation warnings. Technically, the deprecation rev is 0.21, because these apis are removed in 0.22.
Hi Nicolas, for the 0.20 append patch, do you also need to update ClientProtocol?
Patch for 0.20 append looks good, except I don't think we should mark the old APIs as @Deprecated. We'd like to ship 0.20-append as a drop-in replacement for 0.20 for people using HBase, and adding deprecation warnings to 0.20 APIs seems over-zealous.
Backport of patch for 0.20-append branch
Editorial pass over all release notes prior to publication of 0.21.
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #11 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/11/)
Integrated in Hadoop-Hdfs-trunk #85 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/85/)
. Support non-recursive create(). Contributed by Kan Zhang
Integrated in Hadoop-Hdfs-trunk-Commit #36 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/36/)
. Support non-recursive create(). Contributed by Kan Zhang
I have committed this. Thanks, Kan!
Please add release note.
+1 the patch is perfect!
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419664/h617-06.patch
against trunk revision 815001.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/29/console
This message is automatically generated.
attached a new patch that throws FileAlreadyExistsException when parent dir exists as a file.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419593/h617-04.patch
against trunk revision 814449.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/console
This message is automatically generated.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419582/h617-02.patch
against trunk revision 814449.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/console
This message is automatically generated.
If the parent is a file, a FileAlreadyExistsException should be thrown. See also HDFS-602.
one minor javadoc change.
attached a new patch that factors out the code for verifying parent dir exists.
Addressed Nicholas' comment with new patch.
Patch looks good except that createParent should be false when for append.
startFileInternal(src, null, holder, clientMachine, EnumSet.of(CreateFlag.APPEND), - (short)blockManager.maxReplication, (long)0); + true, (short)blockManager.maxReplication, (long)0); getEditLog().logSync();
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419552/h617-01.patch
against trunk revision 814449.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/console
This message is automatically generated.
Atteched a patch that
1. added a boolean parameter to the create() RPC call to indicate whether missing parent directories should be created.
2. added a createNonCursive() method to DistributedFileSystem interface, which doesn't create missing parent directories (complements existing create() method that does).
@Jitendra & @Suresh : since you now have the 3 critical jiras for
HBASE-2312committed to 20-security, it would be good to remove @Ignore from testLogRollAfterSplitStart in HBase to verify that these 3 patches fix HBase lease recovery with code as-is.