Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3804

Both RM are on standBy state when kerberos user not in yarn.admin.acl

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Environment:

      Suse 11 Sp3, 2 RM, Secure

      Description

      Steps to reproduce
      ================
      1. Configure cluster in secure mode
      2. On RM Configure yarn.admin.acl=dsperf
      3. Configure in arn.resourcemanager.principal=yarn
      4. Start Both RM

      Both RM will be in Standby forever

      2015-06-15 12:20:21,556 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn     OPERATION=refreshAdminAcls      TARGET=AdminService     RESULT=FAILURE  DESCRIPTION=Unauthorized userPERMISSIONS=
      2015-06-15 12:20:21,556 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
      org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
              at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
              at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:824)
              at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:420)
              at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:645)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:518)
      Caused by: org.apache.hadoop.ha.ServiceFailedException: Can not execute refreshAdminAcls
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:297)
              at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
              ... 4 more
      Caused by: org.apache.hadoop.yarn.exceptions.YarnException: org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'refreshAdminAcls'
              at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAcls(AdminService.java:230)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAdminAcls(AdminService.java:465)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:295)
              ... 5 more
      Caused by: org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'refreshAdminAcls'
              at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:182)
              at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:148)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(AdminService.java:223)
              at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAcls(AdminService.java:228)
              ... 7 more
      

      Analysis

      On each RM attempt to switch to Active refreshACl is called and acl permission not available for the user
      Infinite retry for the same switch to Active and always false returned from
      ActiveStandbyElector#becomeActive()

      Expected

      RM should get shutdown event after few retry or even at first attempt
      Since at runtime user from which it retries for refreshacl can never be updated.

      States from commands

      ./yarn rmadmin -getServiceState rm2
      standby
      ./yarn rmadmin -getServiceState rm1
      standby

      ./yarn rmadmin -checkHealth rm1
      echo $? = 0
      ./yarn rmadmin -checkHealth rm2
      echo $? = 0

      1. YARN-3804.01.patch
        2 kB
        Varun Saxena
      2. YARN-3804.02.patch
        2 kB
        Varun Saxena
      3. YARN-3804.03.patch
        3 kB
        Varun Saxena
      4. YARN-3804.04.patch
        6 kB
        Varun Saxena
      5. YARN-3804.05.patch
        7 kB
        Varun Saxena
      6. YARN-3804.branch-2.7.patch
        6 kB
        Xuan Gong

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2178 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2178/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2178 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2178/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/230/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/230/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/221/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/221/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2160/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2160/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/232/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/232/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #962 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/962/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #962 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/962/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks for the review and commit Xuan Gong

        Show
        varun_saxena Varun Saxena added a comment - Thanks for the review and commit Xuan Gong
        Hide
        varun_saxena Varun Saxena added a comment -

        Xuan Gong, thanks for updating branch-2.7 patch. Didn't notice your comment due to time difference.

        Show
        varun_saxena Varun Saxena added a comment - Xuan Gong , thanks for updating branch-2.7 patch. Didn't notice your comment due to time difference.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8035 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8035/)
        YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8035 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8035/ ) YARN-3804 . Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena (xgong: rev a826d432f9b45550cc5ab79ef63ca39b176dabb2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
        Hide
        xgong Xuan Gong added a comment -

        Committed into trunk/branch-2/branch-2.7. Thanks, Varun Saxena.

        Show
        xgong Xuan Gong added a comment - Committed into trunk/branch-2/branch-2.7. Thanks, Varun Saxena .
        Hide
        xgong Xuan Gong added a comment -

        Upload a same patch but can apply to branch-2.7

        Show
        xgong Xuan Gong added a comment - Upload a same patch but can apply to branch-2.7
        Hide
        xgong Xuan Gong added a comment -

        Varun Saxena Looks like the patch does not apply for 2.7. Could you provide a patch for branch-2.7, please ?

        Show
        xgong Xuan Gong added a comment - Varun Saxena Looks like the patch does not apply for 2.7. Could you provide a patch for branch-2.7, please ?
        Hide
        xgong Xuan Gong added a comment -

        +1 LGTM. Will commit

        Show
        xgong Xuan Gong added a comment - +1 LGTM. Will commit
        Hide
        varun_saxena Varun Saxena added a comment -

        Test failure unrelated. YARN-3790 already filed for it

        Show
        varun_saxena Varun Saxena added a comment - Test failure unrelated. YARN-3790 already filed for it
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 21m 9s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 10m 22s There were no new javac warning messages.
        +1 javadoc 11m 3s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 53s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 50m 56s Tests failed in hadoop-yarn-server-resourcemanager.
            98m 28s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740038/YARN-3804.05.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5dbc8c9
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8270/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8270/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8270/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 10m 22s There were no new javac warning messages. +1 javadoc 11m 3s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 53s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 50m 56s Tests failed in hadoop-yarn-server-resourcemanager.     98m 28s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740038/YARN-3804.05.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5dbc8c9 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8270/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8270/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8270/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Fixed tests. Moved newly added test case to TestRMAdminService

        Show
        varun_saxena Varun Saxena added a comment - Fixed tests. Moved newly added test case to TestRMAdminService
        Hide
        varun_saxena Varun Saxena added a comment -

        Test failures are related. Will fix them

        Show
        varun_saxena Varun Saxena added a comment - Test failures are related. Will fix them
        Hide
        varun_saxena Varun Saxena added a comment -

        Test failures are related. Will fix them

        Show
        varun_saxena Varun Saxena added a comment - Test failures are related. Will fix them
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 50s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 47s There were no new javac warning messages.
        +1 javadoc 9m 45s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 47s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 49m 32s Tests failed in hadoop-yarn-server-resourcemanager.
            88m 48s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739979/YARN-3804.04.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d4929f4
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8267/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8267/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8267/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8267/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 49m 32s Tests failed in hadoop-yarn-server-resourcemanager.     88m 48s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739979/YARN-3804.04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d4929f4 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8267/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8267/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8267/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8267/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Xuan Gong, added a test case.
        Could not find an appropriate test class to add it in so added it in TestResourceManager.

        Show
        varun_saxena Varun Saxena added a comment - Xuan Gong , added a test case. Could not find an appropriate test class to add it in so added it in TestResourceManager .
        Hide
        varun_saxena Varun Saxena added a comment -

        ok

        Show
        varun_saxena Varun Saxena added a comment - ok
        Hide
        xgong Xuan Gong added a comment -

        Varun Saxena The patch Looks good. But Could we add some testcases for this ?

        Show
        xgong Xuan Gong added a comment - Varun Saxena The patch Looks good. But Could we add some testcases for this ?
        Hide
        varun_saxena Varun Saxena added a comment -

        Anyhow, even that wouldn't have been proper fix because setAdmins is called again on refresh.

        Show
        varun_saxena Varun Saxena added a comment - Anyhow, even that wouldn't have been proper fix because setAdmins is called again on refresh.
        Hide
        varun_saxena Varun Saxena added a comment -

        Refactored the code as per Xuan Gong's suggestion.
        Tested the fix locally on trunk code.

        Show
        varun_saxena Varun Saxena added a comment - Refactored the code as per Xuan Gong 's suggestion. Tested the fix locally on trunk code.
        Hide
        varun_saxena Varun Saxena added a comment -

        Ok, got it...Because we reload the configuration.

        Show
        varun_saxena Varun Saxena added a comment - Ok, got it...Because we reload the configuration.
        Hide
        varun_saxena Varun Saxena added a comment -

        Oops, sorry for the mistake.
        I had tested this in my local setup and removed setAdmins from AdminService. But forgot including changes of AdminService in patch.

        Any reason explicitly calling setAdmins is required by the way ?

        Anyways your suggestion makes sense to avoid issues with another auth provider. Will make the change.

        Show
        varun_saxena Varun Saxena added a comment - Oops, sorry for the mistake. I had tested this in my local setup and removed setAdmins from AdminService. But forgot including changes of AdminService in patch. Any reason explicitly calling setAdmins is required by the way ? Anyways your suggestion makes sense to avoid issues with another auth provider. Will make the change.
        Hide
        xgong Xuan Gong added a comment -

        Varun Saxena
        Actually, in AdminService#serviceInit, we have

            authorizer.setAdmins(new AccessControlList(conf.get(
              YarnConfiguration.YARN_ADMIN_ACL,
                YarnConfiguration.DEFAULT_YARN_ADMIN_ACL)), UserGroupInformation
                .getCurrentUser());
        

        , we could create a common function which will add the Daemon user into the AccessControlList, then pass the modified AccessControlList into this method.
        In this case, we do not need to change the code for every YarnAuthorizationProvider, (such as ConfiguredYarnAuthorizer).

        Also in AdminService#refreshAdminAcls(), we need similar changes, too

        Show
        xgong Xuan Gong added a comment - Varun Saxena Actually, in AdminService#serviceInit, we have authorizer.setAdmins( new AccessControlList(conf.get( YarnConfiguration.YARN_ADMIN_ACL, YarnConfiguration.DEFAULT_YARN_ADMIN_ACL)), UserGroupInformation .getCurrentUser()); , we could create a common function which will add the Daemon user into the AccessControlList, then pass the modified AccessControlList into this method. In this case, we do not need to change the code for every YarnAuthorizationProvider, (such as ConfiguredYarnAuthorizer). Also in AdminService#refreshAdminAcls(), we need similar changes, too
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 29s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
            38m 54s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739897/YARN-3804.02.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b039e69
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8263/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8263/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8263/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 5s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 29s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 33s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.     38m 54s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739897/YARN-3804.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b039e69 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8263/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8263/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8263/console This message was automatically generated.
        Hide
        xgong Xuan Gong added a comment -

        Varun Saxena
        ConfiguredYarnAuthorizer#setAdmins has been called in AdminService.

          @Override
          public void setAdmins(AccessControlList acls, UserGroupInformation ugi) {
            adminAcl = acls;
          }
        

        Could we add the logic here ?

        Show
        xgong Xuan Gong added a comment - Varun Saxena ConfiguredYarnAuthorizer#setAdmins has been called in AdminService. @Override public void setAdmins(AccessControlList acls, UserGroupInformation ugi) { adminAcl = acls; } Could we add the logic here ?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 53s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 47s There were no new javac warning messages.
        +1 javadoc 9m 59s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 53s The applied patch generated 2 new checkstyle issues (total was 2, now 4).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 1m 58s Tests failed in hadoop-yarn-common.
            41m 38s  



        Reason Tests
        Failed unit tests hadoop.yarn.security.TestYARNTokenIdentifier



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739849/YARN-3804.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b039e69
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8260/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8260/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8260/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8260/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 53s The applied patch generated 2 new checkstyle issues (total was 2, now 4). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 1m 58s Tests failed in hadoop-yarn-common.     41m 38s   Reason Tests Failed unit tests hadoop.yarn.security.TestYARNTokenIdentifier Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739849/YARN-3804.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b039e69 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8260/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8260/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8260/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8260/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Bibin A Chundatt, if yarn is the user daemon started with, this will also be fine with the patch submitted. If this is not same as the daemon user, you will have to configure it.

        Show
        varun_saxena Varun Saxena added a comment - Bibin A Chundatt , if yarn is the user daemon started with, this will also be fine with the patch submitted. If this is not same as the daemon user, you will have to configure it.
        Hide
        varun_saxena Varun Saxena added a comment -

        Added the user which daemon starts with, in the list of Admin ACLs', so that it matches.

        Show
        varun_saxena Varun Saxena added a comment - Added the user which daemon starts with, in the list of Admin ACLs', so that it matches.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Varun Saxena ,Vinod Kumar Vavilapalli Also at start up getServiceState will throw exception

        2015-06-16 16:48:38,246 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn	IP=10.19.92.128	OPERATION=getServiceState	TARGET=AdminService	RESULT=FAILURE	DESCRIPTION=Unauthorized user	PERMISSIONS=
        2015-06-16 16:48:38,247 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 45021, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from 10.19.92.128:53773 Call#238 Retry#0
        org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'getServiceState'
        	at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:182)
        	at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:148)
        	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(AdminService.java:223)
        	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.getServiceStatus(AdminService.java:344)
        	at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.getServiceStatus(HAServiceProtocolServerSideTranslatorPB.java:131)
        	at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4464)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:972)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2088)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2084)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:422)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672)
        	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2082)
        2015-06-16 16:48:38,258 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user yarn
        
        

        Should we handle this too??

        Show
        bibinchundatt Bibin A Chundatt added a comment - Varun Saxena , Vinod Kumar Vavilapalli Also at start up getServiceState will throw exception 2015-06-16 16:48:38,246 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn IP=10.19.92.128 OPERATION=getServiceState TARGET=AdminService RESULT=FAILURE DESCRIPTION=Unauthorized user PERMISSIONS= 2015-06-16 16:48:38,247 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 45021, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from 10.19.92.128:53773 Call#238 Retry#0 org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'getServiceState' at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:182) at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:148) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(AdminService.java:223) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.getServiceStatus(AdminService.java:344) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.getServiceStatus(HAServiceProtocolServerSideTranslatorPB.java:131) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4464) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:972) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2088) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2084) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2082) 2015-06-16 16:48:38,258 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user yarn Should we handle this too??
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Vinod Kumar Vavilapalli

        Allow the daemon user to do the refresh irrespective of what admin configures

        sounds better to me.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Vinod Kumar Vavilapalli Allow the daemon user to do the refresh irrespective of what admin configures sounds better to me.
        Hide
        kasha Karthik Kambatla added a comment -

        On board with suggestions here.

        Show
        kasha Karthik Kambatla added a comment - On board with suggestions here.
        Hide
        leftnoteasy Wangda Tan added a comment -

        There's a inconsistent check in current code path:

        • AdminService.checkAccess uses YarnAuthorizationProvider to do the check. In its default implementation: ConfiguredYarnAuthorizer, it uses configured yarn.admin.acl
        • ClientRMService.checkAccess uses AdminCLIsManager, it uses configured yarn.admin.acl + daemon_user

        I think we should fix the inconsistency issue, 2) will be completed with if we make both of them allow daemont_user.

        Show
        leftnoteasy Wangda Tan added a comment - There's a inconsistent check in current code path: AdminService.checkAccess uses YarnAuthorizationProvider to do the check. In its default implementation: ConfiguredYarnAuthorizer , it uses configured yarn.admin.acl ClientRMService.checkAccess uses AdminCLIsManager, it uses configured yarn.admin.acl + daemon_user I think we should fix the inconsistency issue, 2) will be completed with if we make both of them allow daemont_user .
        Hide
        xgong Xuan Gong added a comment -

        I am OK with that.
        In transitionToActive(), we are re-using all the refresh* code, if we choose option 2, we need to re-factory all the refresh* functions.

        Show
        xgong Xuan Gong added a comment - I am OK with that. In transitionToActive(), we are re-using all the refresh* code, if we choose option 2, we need to re-factory all the refresh* functions.
        Hide
        jianhe Jian He added a comment -

        +1 for 2)
        Not too much point having RM to depend on the admin acl to do transition for itself. Karthik Kambatla, Xuan Gong, sounds good ?

        Show
        jianhe Jian He added a comment - +1 for 2) Not too much point having RM to depend on the admin acl to do transition for itself. Karthik Kambatla , Xuan Gong , sounds good ?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Seems like a critical issue to me.

        Two options

        1. Fail correctly and assume that admin adds yarn user explicitly if it needs to work.
        2. Allow the daemon user to do the refresh irrespective of what admin configures

        I get a feeling (2) is better. Thoughts? /cc Wangda Tan, Jian He

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Seems like a critical issue to me. Two options Fail correctly and assume that admin adds yarn user explicitly if it needs to work. Allow the daemon user to do the refresh irrespective of what admin configures I get a feeling (2) is better. Thoughts? /cc Wangda Tan , Jian He
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Can we check for AccessControlException in ActiveStandbyElector#becomeActive() send event to shutdown ?

        Show
        bibinchundatt Bibin A Chundatt added a comment - Can we check for AccessControlException in ActiveStandbyElector#becomeActive() send event to shutdown ?

          People

          • Assignee:
            varun_saxena Varun Saxena
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development