Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: hdfs-client, namenode
    • Labels:
      None

      Description

      HDFS permisson check is incorrect, even if dfs.permissions is set false. it does look like this was caused by snap shot.

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          it does look like this was caused by snap shot.

          Affects Version/s is set to 2.0.4-alpha. Snapshot feature is not in that release. Which release did you see this problem in?

          Also can you add more details what you mean by permission check is incorrect.

          Show
          Suresh Srinivas added a comment - it does look like this was caused by snap shot. Affects Version/s is set to 2.0.4-alpha. Snapshot feature is not in that release. Which release did you see this problem in? Also can you add more details what you mean by permission check is incorrect.
          Hide
          Harsh J added a comment -

          The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. The user thread for this report is http://search-hadoop.com/m/r91Yo1soPIp.

          Show
          Harsh J added a comment - The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. The user thread for this report is http://search-hadoop.com/m/r91Yo1soPIp .
          Hide
          Fengdong Yu added a comment -

          I see this problem from trunk.

          HDFS operation works well, but when submit MR, I always get the following:

          ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: The ownership/permissions on the staging directory hdfs://webdm-cluster/data/hadoop/data/mapred/staging/root/.staging is not as expected. It is owned by root and permissions are rwxr-xr-x. The directory must be owned by the submitter root or by root and permissions must be rwx------
          
          

          chown and chmod cannot solve the problem.

          so I read the code again. and get the following:
          FSPermissionChecker.java:

           void checkPermission(String path, INodeDirectory root, boolean doCheckOwner,
                FsAction ancestorAccess, FsAction parentAccess, FsAction access,
                FsAction subAccess, boolean resolveLink)
             final Snapshot snapshot = inodesInPath.getPathSnapshot();
          

          then:

            /** Guarded by {@link FSNamesystem#readLock()} */
            private void check(INode inode, Snapshot snapshot, FsAction access
                ) throws AccessControlException {
              if (inode == null) {
                return;
              }
              FsPermission mode = inode.getFsPermission(snapshot);
          
              if (user.equals(inode.getUserName(snapshot))) { //user class
                if (mode.getUserAction().implies(access)) { return; }
              }
              else if (groups.contains(inode.getGroupName(snapshot))) { //group class
                if (mode.getGroupAction().implies(access)) { return; }
              }
              else { //other class
                if (mode.getOtherAction().implies(access)) { return; }
              }
              throw new AccessControlException("Permission denied: user=" + user
                  + ", access=" + access + ", inode=" + toAccessControlString(inode));
            }
          

          so , If snapshot is null, all permission check are all did by "other class".

          Show
          Fengdong Yu added a comment - I see this problem from trunk. HDFS operation works well, but when submit MR, I always get the following: ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: The ownership/permissions on the staging directory hdfs: //webdm-cluster/data/hadoop/data/mapred/staging/root/.staging is not as expected. It is owned by root and permissions are rwxr-xr-x. The directory must be owned by the submitter root or by root and permissions must be rwx------ chown and chmod cannot solve the problem. so I read the code again. and get the following: FSPermissionChecker.java: void checkPermission( String path, INodeDirectory root, boolean doCheckOwner, FsAction ancestorAccess, FsAction parentAccess, FsAction access, FsAction subAccess, boolean resolveLink) final Snapshot snapshot = inodesInPath.getPathSnapshot(); then: /** Guarded by {@link FSNamesystem#readLock()} */ private void check(INode inode, Snapshot snapshot, FsAction access ) throws AccessControlException { if (inode == null ) { return ; } FsPermission mode = inode.getFsPermission(snapshot); if (user.equals(inode.getUserName(snapshot))) { //user class if (mode.getUserAction().implies(access)) { return ; } } else if (groups.contains(inode.getGroupName(snapshot))) { //group class if (mode.getGroupAction().implies(access)) { return ; } } else { //other class if (mode.getOtherAction().implies(access)) { return ; } } throw new AccessControlException( "Permission denied: user=" + user + ", access=" + access + ", inode=" + toAccessControlString(inode)); } so , If snapshot is null, all permission check are all did by "other class".
          Hide
          Fengdong Yu added a comment -

          If I am right, I can work on it for a patch.

          Show
          Fengdong Yu added a comment - If I am right, I can work on it for a patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > so, If snapshot is null, all permission check are all did by "other class".

          Why? When snapshot is null, inode.getUserName(snapshot) returns the current owner of the inode.

          Show
          Tsz Wo Nicholas Sze added a comment - > so, If snapshot is null, all permission check are all did by "other class". Why? When snapshot is null, inode.getUserName(snapshot) returns the current owner of the inode.
          Hide
          Fengdong Yu added a comment -

          The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. The user thread for this report is http://search-hadoop.com/m/r91Yo1soPIp.

          I enabled permission in my test cluster. because "dfs.permissions.enabled" is true by default. then I cannot submit MR,
          so I don't think NN RPC call doesn't check "dfs.permissions.enabled". and I checked code again. It does check.

          after submit MR, found on the job tracker.

          Failure Info: Job initialization failed: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=EXECUTE, inode="/data/hadoop/data/mapred/system":hadoop:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:191) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:154) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4848) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4830) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4804) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1927) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1827) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:459) at 
          
          Show
          Fengdong Yu added a comment - The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. The user thread for this report is http://search-hadoop.com/m/r91Yo1soPIp . I enabled permission in my test cluster. because "dfs.permissions.enabled" is true by default. then I cannot submit MR, so I don't think NN RPC call doesn't check "dfs.permissions.enabled". and I checked code again. It does check. after submit MR, found on the job tracker. Failure Info: Job initialization failed: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=EXECUTE, inode= "/data/hadoop/data/mapred/system" :hadoop:supergroup:drwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:191) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:154) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4848) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4830) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4804) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1927) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1827) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:459) at
          Hide
          Prashant Kommireddi added a comment -

          Agree with Harsh, setPermissions needs to honor isPermissionsEnabled. This explanation might provide some more context to how it affects MR jobs (not really related to snapshots) http://search-hadoop.com/m/bKzZK1soPIp

          Show
          Prashant Kommireddi added a comment - Agree with Harsh, setPermissions needs to honor isPermissionsEnabled. This explanation might provide some more context to how it affects MR jobs (not really related to snapshots) http://search-hadoop.com/m/bKzZK1soPIp
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. ...

          If permission is disabled, user should not call setPermission/setOwner since setting them does not have the desired effect. Perhaps we should throw some other exceptions such as IllegalStateException.

          Show
          Tsz Wo Nicholas Sze added a comment - > The issue (I don't think its snapshot related either) is that the NN RPC call of setPermission does not check the state of isPermissionsEnabled. ... If permission is disabled, user should not call setPermission/setOwner since setting them does not have the desired effect. Perhaps we should throw some other exceptions such as IllegalStateException.
          Hide
          Prashant Kommireddi added a comment -

          > user should not call setPermission/setOwner since setting them does not have the desired effect.

          Just wanted to highlight, I think it's the framework that is actually making a call on setPermission when a MR job starts and not the user explicitly.

          Show
          Prashant Kommireddi added a comment - > user should not call setPermission/setOwner since setting them does not have the desired effect. Just wanted to highlight, I think it's the framework that is actually making a call on setPermission when a MR job starts and not the user explicitly.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... This explanation might provide some more context to how it affects MR jobs ...

          Out of my curiosity, why disable permission in your cluster? I beg most existing libraries such as the MR code assume permission is enabled and do not handle the case that permission is disabled.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... This explanation might provide some more context to how it affects MR jobs ... Out of my curiosity, why disable permission in your cluster? I beg most existing libraries such as the MR code assume permission is enabled and do not handle the case that permission is disabled.
          Hide
          Prashant Kommireddi added a comment -

          We don't have very tight security requirements on our cluster at this point, and it felt easier to disable the perms for now (that is how we also operated 0.20.2). In this case specifically, I don't think enabling perms would make a difference as there would be an issue with another user (other than the one that started the server) from writing to some directories?

          I guess the question is more about backward-compatibility and also if the feature (dfs.permissions.enabled) is actually expected to behave as its documented. We can turn on perms if required, but the issue still remains.

          Show
          Prashant Kommireddi added a comment - We don't have very tight security requirements on our cluster at this point, and it felt easier to disable the perms for now (that is how we also operated 0.20.2). In this case specifically, I don't think enabling perms would make a difference as there would be an issue with another user (other than the one that started the server) from writing to some directories? I guess the question is more about backward-compatibility and also if the feature (dfs.permissions.enabled) is actually expected to behave as its documented. We can turn on perms if required, but the issue still remains.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I guess the question is more about backward-compatibility and also if the feature (dfs.permissions.enabled) is actually expected to behave as its documented. We can turn on perms if required, but the issue still remains.

          Just checked the branch-1 code, it is the same as trunk: setPermission(..)/setOwner(..) does not check isPermissionsEnabled; see http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java?view=annotate

          That part of HDFS code has not been changed for a long time. So there is no compatibility issue in HDFS. You may want to file a MR jira.

          Also checked the documentation, it says that the current behavior is correct, i.e. setPermission(..)/setOwner(..) should always check permission. Below is quoted from http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Configuration_Parameters

          ... Regardless of whether permissions are on or off, chmod, chgrp and chown always check permissions. These functions are only useful in the permissions context, and so there is no backwards compatibility issue. Furthermore, this allows administrators to reliably set owners and permissions in advance of turning on regular permissions checking.

          Show
          Tsz Wo Nicholas Sze added a comment - > I guess the question is more about backward-compatibility and also if the feature (dfs.permissions.enabled) is actually expected to behave as its documented. We can turn on perms if required, but the issue still remains. Just checked the branch-1 code, it is the same as trunk: setPermission(..)/setOwner(..) does not check isPermissionsEnabled; see http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java?view=annotate That part of HDFS code has not been changed for a long time. So there is no compatibility issue in HDFS. You may want to file a MR jira. Also checked the documentation, it says that the current behavior is correct, i.e. setPermission(..)/setOwner(..) should always check permission. Below is quoted from http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Configuration_Parameters ... Regardless of whether permissions are on or off, chmod, chgrp and chown always check permissions. These functions are only useful in the permissions context, and so there is no backwards compatibility issue. Furthermore, this allows administrators to reliably set owners and permissions in advance of turning on regular permissions checking.
          Hide
          Fengdong Yu added a comment -

          for exactly, My question is similar to this issue:
          https://issues.apache.org/jira/browse/MAPREDUCE-5112

          Show
          Fengdong Yu added a comment - for exactly, My question is similar to this issue: https://issues.apache.org/jira/browse/MAPREDUCE-5112
          Hide
          Fengdong Yu added a comment -

          This was caused by fair-scheduler, not HDFS. Thanks.

          https://issues.apache.org/jira/browse/MAPREDUCE-4451

          This was fixed in hadoop-1.2.0, then, should we close this issue?

          Show
          Fengdong Yu added a comment - This was caused by fair-scheduler, not HDFS. Thanks. https://issues.apache.org/jira/browse/MAPREDUCE-4451 This was fixed in hadoop-1.2.0, then, should we close this issue?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Fengdong, thanks for checking it. Closing this.

          Show
          Tsz Wo Nicholas Sze added a comment - Fengdong, thanks for checking it. Closing this.
          Hide
          Fengdong Yu added a comment -

          Add some addtional to avoid confusing, My testing cluster using HDFSv2,Commonv2 from trunk and Mapreducev1 from 1.1.2, so I can confirm this issue is fair scheduler caused.

          but if MR is also from the trunk, then I am not sure the root cause is fair scheduler.

          Show
          Fengdong Yu added a comment - Add some addtional to avoid confusing, My testing cluster using HDFSv2,Commonv2 from trunk and Mapreducev1 from 1.1.2, so I can confirm this issue is fair scheduler caused. but if MR is also from the trunk, then I am not sure the root cause is fair scheduler.
          Hide
          Prashant Kommireddi added a comment -

          I am not using the FairScheduler, so the root cause must not be that. Tsz Wo Nicholas Sze Harsh J you guys feel this should be a MR issue?

          Show
          Prashant Kommireddi added a comment - I am not using the FairScheduler, so the root cause must not be that. Tsz Wo Nicholas Sze Harsh J you guys feel this should be a MR issue?
          Hide
          Chris Nauroth added a comment -

          Also checked the documentation, it says that the current behavior is correct

          Since this caused confusion for a few of us, I filed HDFS-4919 to improve the documentation. The first place I look for an explanation of a configuration property is hdfs-default.xml, and the description of dfs.permissions.enabled in there doesn't mention that permissions are still checked for some operations regardless of this setting.

          Also, the HDFS permissions guide doc still mentions the deprecated dfs.permissions property instead of the newer dfs.permissions.enabled.

          Show
          Chris Nauroth added a comment - Also checked the documentation, it says that the current behavior is correct Since this caused confusion for a few of us, I filed HDFS-4919 to improve the documentation. The first place I look for an explanation of a configuration property is hdfs-default.xml, and the description of dfs.permissions.enabled in there doesn't mention that permissions are still checked for some operations regardless of this setting. Also, the HDFS permissions guide doc still mentions the deprecated dfs.permissions property instead of the newer dfs.permissions.enabled.

            People

            • Assignee:
              Unassigned
              Reporter:
              Fengdong Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development