Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12413

AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Currently AccessControlList will call ugi.getGroupNames() in isUserInList even if groups is empty. ugi.getGroupNames() is an expensive operation which call shell script id -gn <USER> && id -Gn <user> to get the list of groups. For example,
      ServiceAuthorizationManager#authorize will call blocked ACL acls[1].isUserAllowed(user) to check the user permission. The default value for blocked ACL is empty

          String defaultBlockedAcl = conf.get(   CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_AUTHORIZATION_DEFAULT_BLOCKED_ACL, "");
      

      So every time authorize is called, getGroupNames may be called.
      It also caused the following warning message:

      2015-09-08 14:55:34,236 WARN [Socket Reader #1 for port 52715] org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user job_1441722221553_0005: id: job_1441722221553_0005: No such user
      2015-09-08 14:55:34,236 WARN [Socket Reader #1 for port 52715] org.apache.hadoop.security.UserGroupInformation: No groups available for user job_1441722221553_0005
      2015-09-08 14:55:34,236 INFO [Socket Reader #1 for port 52715] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1441722221553_0005 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
      

        Issue Links

          Activity

          Hide
          zxu zhihai xu added a comment -

          I attached a patch HADOOP-12413.000.patch which skip calling ugi.getGroupNames() if groups is empty.

          Show
          zxu zhihai xu added a comment - I attached a patch HADOOP-12413 .000.patch which skip calling ugi.getGroupNames() if groups is empty.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 22m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 10m 22s There were no new javac warning messages.
          +1 javadoc 13m 13s There were no new javadoc warning messages.
          +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 2m 14s mvn install still works.
          +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse.
          +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 27m 16s Tests failed in hadoop-common.
              80m 34s  



          Reason Tests
          Failed unit tests hadoop.fs.TestLocalFsFCStatistics
            hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12755908/HADOOP-12413.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d777757
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 22m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 22s There were no new javac warning messages. +1 javadoc 13m 13s There were no new javadoc warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 2m 14s mvn install still works. +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse. +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 27m 16s Tests failed in hadoop-common.     80m 34s   Reason Tests Failed unit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755908/HADOOP-12413.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d777757 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7662/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Nice find, zhihai xu! +1 for the patch. I confirmed that the test failures were unrelated. I have committed this to trunk and branch-2.

          Show
          cnauroth Chris Nauroth added a comment - Nice find, zhihai xu ! +1 for the patch. I confirmed that the test failures were unrelated. I have committed this to trunk and branch-2.
          Hide
          zxu zhihai xu added a comment -

          Chris Nauroth, thanks for the review and committing the patch!

          Show
          zxu zhihai xu added a comment - Chris Nauroth , thanks for the review and committing the patch!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1128 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1128/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1128 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1128/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 14s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 10m 9s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 7s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 23m 1s Tests failed in hadoop-common.
              63m 48s  



          Reason Tests
          Failed unit tests hadoop.ipc.TestRPC
          Timed out tests org.apache.hadoop.ipc.TestRPCCompatibility



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12756033/HADOOP-12413.000.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 083b44c
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 9s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 7s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 23m 1s Tests failed in hadoop-common.     63m 48s   Reason Tests Failed unit tests hadoop.ipc.TestRPC Timed out tests org.apache.hadoop.ipc.TestRPCCompatibility Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12756033/HADOOP-12413.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 083b44c hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7663/console This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8455 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8455/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8455 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8455/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #389 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/389/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #389 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/389/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #395 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/395/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #395 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/395/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2337 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2337/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2337 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2337/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/374/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #374 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/374/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2314 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2314/)
          HADOOP-12413. AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2314 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2314/ ) HADOOP-12413 . AccessControlList should avoid calling getGroupNames in isUserInList with empty groups. Contributed by Zhihai Xu. (cnauroth: rev b2017d9b032af20044fdf60ddbd1575a554ccb79) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/authorize/TestAccessControlList.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/AccessControlList.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, zhihai xu! I light of the problems reported in YARN-3452 and YARN-4336, I pulled this into branch-2.7 and branch-2.6. This should avoid the bogus user lookups for those that aren't using the reverse-ACL feature.

          Show
          jlowe Jason Lowe added a comment - Thanks, zhihai xu ! I light of the problems reported in YARN-3452 and YARN-4336 , I pulled this into branch-2.7 and branch-2.6. This should avoid the bogus user lookups for those that aren't using the reverse-ACL feature.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              zxu zhihai xu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development