Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3081

The static method FileSystem.mkdirs() should use the correct permissions to create directory

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Invalid
    • 0.16.1
    • None
    • None
    • None

    Description

      There is a static method FileSystem.mkdirs() that create the directory first and then sets permissions in it. Instead, it should create the directory with the correct permissions.

      Otherwise, even if dfs.permission is set to off, one might see job submission failures with the following stack trace:

      at org.apache.hadoop.dfs.PermissionChecker.check(PermissionChecker.java:173)
      at
      org.apache.hadoop.dfs.PermissionChecker.checkTraverse(PermissionChecker.java:129)
      at
      org.apache.hadoop.dfs.PermissionChecker.checkPermission(PermissionChecker.java:99)
      at org.apache.hadoop.dfs.FSNamesystem.checkPermission(FSNamesystem.java:4031)
      at org.apache.hadoop.dfs.FSNamesystem.checkOwner(FSNamesystem.java:3986)
      at org.apache.hadoop.dfs.FSNamesystem.setPermission(FSNamesystem.java:715)
      at org.apache.hadoop.dfs.NameNode.setPermission(NameNode.java:297)
      at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
      at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:899)

      at org.apache.hadoop.ipc.Client.call(Client.java:512)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
      at org.apache.hadoop.dfs.$Proxy4.setPermission(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at
      org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at
      org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at org.apache.hadoop.dfs.$Proxy4.setPermission(Unknown Source)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:351)
      at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:218)
      at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:688)
      at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:124)
      at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2114)

      Attachments

        Issue Links

          Activity

            cutting Doug Cutting added a comment -

            The static method is required to avoid umask, as is sometimes required, yet still have mkdirs implement POSIX conventions. So calling mkdirs(path, permissions) won't do what's desired here, will it?

            I don't see the exception in your stack trace. How does it fail?

            cutting Doug Cutting added a comment - The static method is required to avoid umask, as is sometimes required, yet still have mkdirs implement POSIX conventions. So calling mkdirs(path, permissions) won't do what's desired here, will it? I don't see the exception in your stack trace. How does it fail?
            szetszwo Tsz-wo Sze added a comment -

            The non-static FileSystem.mkdirs(Path f, FsPermission permission) and the static FileSystem.mkdirs(FileSystem fs, Path dir, FsPermission permission) look similar but they are different in semantic:

            • The non-static FileSystem.mkdirs(Path f, FsPermission permission) likes mkdir in Unix, it makes a directory with umask.
            • However, the static FileSystem.mkdirs(FileSystem fs, Path dir, FsPermission permission) makes a directory with absolute permission.

            So I think it is better to change one of the method name (probably the static one), otherwise, it would be confusing.

            szetszwo Tsz-wo Sze added a comment - The non-static FileSystem.mkdirs(Path f, FsPermission permission) and the static FileSystem.mkdirs(FileSystem fs, Path dir, FsPermission permission) look similar but they are different in semantic: The non-static FileSystem.mkdirs(Path f, FsPermission permission) likes mkdir in Unix, it makes a directory with umask . However, the static FileSystem.mkdirs(FileSystem fs, Path dir, FsPermission permission) makes a directory with absolute permission . So I think it is better to change one of the method name (probably the static one), otherwise, it would be confusing.

            Is there a reason you don't just jettison the static one and use mkdir() followed by a chmod() like UNIX? The static one will break setuid/setgid if/when it is supported.

            aw Allen Wittenauer added a comment - Is there a reason you don't just jettison the static one and use mkdir() followed by a chmod() like UNIX? The static one will break setuid/setgid if/when it is supported.
            cutting Doug Cutting added a comment -

            > jettison the static one and use mkdir() followed by a chmod() like UNIX?

            That's exactly what the static implementation does. It's just a utility that saves applications a few lines of code.

            cutting Doug Cutting added a comment - > jettison the static one and use mkdir() followed by a chmod() like UNIX? That's exactly what the static implementation does. It's just a utility that saves applications a few lines of code.

            People

              hairong Hairong Kuang
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: