Hadoop Common
  1. Hadoop Common
  2. HADOOP-6897

FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.

        public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
        throws IOException {
          // create the directory using the default permission
          boolean result = fs.mkdirs(dir);
          // set its permission to be the supplied one
          fs.setPermission(dir, permission);
          return result;
        }
      
      1. mkdirs.patch
        0.8 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          I second Nicholas' comment:

          I think that the static mkdirs(..) is confusing to users. There are only a few lines of codes inside. Why don't we just deprecate it now and remove it in the future?

          This should already be covered by the FileSystem#mkdir/mkdirs methods now. I do not see why we should have a static method for mkdirs and create anymore.

          Show
          Harsh J added a comment - I second Nicholas' comment: I think that the static mkdirs(..) is confusing to users. There are only a few lines of codes inside. Why don't we just deprecate it now and remove it in the future? This should already be covered by the FileSystem#mkdir/mkdirs methods now. I do not see why we should have a static method for mkdirs and create anymore.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch
          against trunk revision 1071364.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/265//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch
          against trunk revision 1031422.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/26//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Map/Reduce could just replace it with two or three lines of codes.

          I forgot to mention that there is another drawback about the static mkdir(..). Some users calling it do not realize that it is actually more expensive compared with the non-static mkdir(..) – it requires one more rpc calls.

          Show
          Tsz Wo Nicholas Sze added a comment - Map/Reduce could just replace it with two or three lines of codes. I forgot to mention that there is another drawback about the static mkdir(..). Some users calling it do not realize that it is actually more expensive compared with the non-static mkdir(..) – it requires one more rpc calls.
          Hide
          Hairong Kuang added a comment -

          This utility is used by Map/Reduce. So removing it may not be an option.

          Show
          Hairong Kuang added a comment - This utility is used by Map/Reduce. So removing it may not be an option.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I think that the static mkdirs(..) is confusing to users. There are only a few lines of codes inside. Why don't we just deprecate it now and remove it in the future?

          In the meantime, we could update the javadoc as follows:

          /** 
           * This method is equivalent to the following codes:
           *
           *    if (fs.mkdirs(dir)) {
           *      fs.setPermission(dir, permission);
           *      return true;
           *    }
           *    return false;
           */
          
          Show
          Tsz Wo Nicholas Sze added a comment - I think that the static mkdirs(..) is confusing to users. There are only a few lines of codes inside. Why don't we just deprecate it now and remove it in the future? In the meantime, we could update the javadoc as follows: /** * This method is equivalent to the following codes: * * if (fs.mkdirs(dir)) { * fs.setPermission(dir, permission); * return true ; * } * return false ; */
          Hide
          Hairong Kuang added a comment -

          Nicholas, shall we discuss the expected behavior of the static mkdirs method? At least I think it should be consistent with the regular mkdirs except that it sets the absolute permission.

          Show
          Hairong Kuang added a comment - Nicholas, shall we discuss the expected behavior of the static mkdirs method? At least I think it should be consistent with the regular mkdirs except that it sets the absolute permission.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Hairong, could you also clarify javadoc for the expected behavior of public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Hairong, could you also clarify javadoc for the expected behavior of public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)?
          Hide
          Hairong Kuang added a comment -

          Dick, you made a good observation. But my understanding of their semantics is a little different from yours.

          I believe static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission) assigns masked permission to intermediate directories, if any, that get created, and assigns the given permission to the leaf directory.

          On the other hand fs.mkdirs(path, permission) assign the masked permission to all newly created directories.

          Another subtle difference is that if the directory already exists, static boolean FileSystem mkdirs would go ahead and change the directory's permission. But boolean fs.mkdirs would simply return true without changing its permission.

          Show
          Hairong Kuang added a comment - Dick, you made a good observation. But my understanding of their semantics is a little different from yours. I believe static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission) assigns masked permission to intermediate directories, if any, that get created, and assigns the given permission to the leaf directory. On the other hand fs.mkdirs(path, permission) assign the masked permission to all newly created directories. Another subtle difference is that if the directory already exists, static boolean FileSystem mkdirs would go ahead and change the directory's permission. But boolean fs.mkdirs would simply return true without changing its permission.
          Hide
          Dick King added a comment -

          There's a wierd incompatibility within the various overloads of FileSystem.mkdirs .

          static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission) assigns the specified permission to all directories constructed on the way to path , including the last one, path itself .

          fs.mkdirs(path, permission) behaves very differently. The intermediate directories, if any, that get created, are born with the permissions of permission , but the leaf directory, path , seems to get its permissions set to a MASKED permission mask, at least when FileSystem is a DFS.

          At the very least, this behavior should be documented. A better remedy, in my opinion, would be for FileSystem.mkdirs(Path, FsPermission) to change behavior so all directories created have permission bits named by the FsPermission parameter in all of the calls that have one.

          Show
          Dick King added a comment - There's a wierd incompatibility within the various overloads of FileSystem.mkdirs . static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission) assigns the specified permission to all directories constructed on the way to path , including the last one, path itself . fs.mkdirs(path, permission) behaves very differently. The intermediate directories, if any, that get created, are born with the permissions of permission , but the leaf directory, path , seems to get its permissions set to a MASKED permission mask, at least when FileSystem is a DFS. At the very least, this behavior should be documented. A better remedy, in my opinion, would be for FileSystem.mkdirs(Path, FsPermission) to change behavior so all directories created have permission bits named by the FsPermission parameter in all of the calls that have one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch
          against trunk revision 981714.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch against trunk revision 981714. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          The fix is trivial. So no unit test is included.

          Show
          Hairong Kuang added a comment - The fix is trivial. So no unit test is included.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development