Hadoop Common
  1. Hadoop Common
  2. HADOOP-6271

Fix FileContext to allow both recursive and non recursive create and mkdir

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Modify FileContext to allow recursive and non-recursive create and mkdir (see HADOOP-4952)

      1. mkdirCreate1.patch
        38 kB
        Sanjay Radia
      2. mkdirCreate2.patch
        38 kB
        Sanjay Radia
      3. mkdirCreate3.patch
        39 kB
        Sanjay Radia
      4. mkdirCreate7.patch
        32 kB
        Sanjay Radia
      5. mkdirCreate8.patch
        32 kB
        Sanjay Radia

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420032/mkdirCreate1.patch
          against trunk revision 816532.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420032/mkdirCreate1.patch against trunk revision 816532. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/50/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420043/mkdirCreate3.patch
          against trunk revision 816703.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420043/mkdirCreate3.patch against trunk revision 816703. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/51/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          nits:

          • In FileContextCreateMkdirBaseTest.java, ClusterShutdownAtEnd() -> clusterShutdownAtEnd()
          • In FileContext.java, extra empty line in javadoc for create()

          code:

          • In FileSystem.java, in primitiveCreate(), the following should throw FileAlreadyExistsException. Same in primitiveMkdir().
            +      if (!stat.isDir()) {
            +          throw new FileNotFoundException("parent is not a dir:" + f);
            +      }
            
          • In FileContext#create() method, newOpts is constructed with (opts.length +1) to accommodate a new permission option being added to opts, which means there will be two permission options in newOpts. This will result in IllegalArgumentException("multiple varargs of same kind") in the underlying layer. Would it be better to construct newOpts with the same length as opts and set the permission option in newOpts to the new one as you parse opts?
          Show
          Kan Zhang added a comment - nits: In FileContextCreateMkdirBaseTest.java, ClusterShutdownAtEnd() -> clusterShutdownAtEnd() In FileContext.java, extra empty line in javadoc for create() code: In FileSystem.java, in primitiveCreate(), the following should throw FileAlreadyExistsException. Same in primitiveMkdir(). + if (!stat.isDir()) { + throw new FileNotFoundException( "parent is not a dir:" + f); + } In FileContext#create() method, newOpts is constructed with (opts.length +1) to accommodate a new permission option being added to opts, which means there will be two permission options in newOpts. This will result in IllegalArgumentException("multiple varargs of same kind") in the underlying layer. Would it be better to construct newOpts with the same length as opts and set the permission option in newOpts to the new one as you parse opts?
          Hide
          Kan Zhang added a comment -

          > Would it be better to construct newOpts with the same length as opts and set the permission option in newOpts to the new one as you parse opts?

          Sorry, I missed the case where permission is not set in the original opts. However, it may result in IllegalArgumentException("multiple varargs of same kind") in case it is set, right?

          Show
          Kan Zhang added a comment - > Would it be better to construct newOpts with the same length as opts and set the permission option in newOpts to the new one as you parse opts? Sorry, I missed the case where permission is not set in the original opts. However, it may result in IllegalArgumentException("multiple varargs of same kind") in case it is set, right?
          Hide
          Sanjay Radia added a comment -

          @Kan - latest patch addresses yours and Jokob's comments he gave me offline.

          Show
          Sanjay Radia added a comment - @Kan - latest patch addresses yours and Jokob's comments he gave me offline.
          Hide
          Jakob Homan added a comment -

          After updates, looks good. +1

          Show
          Jakob Homan added a comment - After updates, looks good. +1
          Hide
          Kan Zhang added a comment -

          +1.

          Show
          Kan Zhang added a comment - +1.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420107/mkdirCreate8.patch
          against trunk revision 816794.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420107/mkdirCreate8.patch against trunk revision 816794. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/15/console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #43 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/43/)
          . Add recursive and non recursive create and mkdir to FileContext. Contributed by Sanjay Radia.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #43 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/43/ ) . Add recursive and non recursive create and mkdir to FileContext. Contributed by Sanjay Radia.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #102 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/102/)
          . Add recursive and non recursive create and mkdir to FileContext. Contributed by Sanjay Radia.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #102 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/102/ ) . Add recursive and non recursive create and mkdir to FileContext. Contributed by Sanjay Radia.
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21. Just one FC item.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21. Just one FC item.

            People

            • Assignee:
              Sanjay Radia
              Reporter:
              Sanjay Radia
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development