Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
trunk
-
None
Description
This issue was reported by kbadani.
Steps to reproduce this issue:
- Kdestroy and kinit as 'livy' user
- Do spark-submit with --proxy-user as 'hrt_1'
- In the console output, you can see that 'ambari-qa' is trying to impersonate as 'hrt_1' and its failing
- Cancel the running job and do klist again - it will show credentials for 'ambari-qa' user and not the 'livy' user with which it was kinited
[livy@ctr-e133-1493418528701-6489-01-000003 spark]$ kinit -kt /etc/security/keytabs/livy.service.keytab livy/ctr-e133-1493418528701-6489-01-000003.hwx.site [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist Ticket cache: FILE:/tmp/krb5cc_1808 Default principal: livy/ctr-e133-1493418528701-6489-01-000003.hwx.site@EXAMPLE.COM Valid starting Expires Service principal 05/02/2017 23:52:14 05/03/2017 23:52:14 krbtgt/EXAMPLE.COM@EXAMPLE.COM [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 512m --executor-memory 512m --proxy-user hrt_1 --executor-cores 1 /usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-45-hadoop2.7.3.2.6.1.0-45.jar 10 Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default 17/05/02 23:53:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/05/02 23:53:12 INFO AHSProxy: Connecting to Application History server at ctr-e133-1493418528701-6489-01-000004.hwx.site/172.27.22.136:10200 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]... 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation returned exception on [rm1] : org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, so propagating back to caller. 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Connection lost with rm1, trying to fail over. 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]... 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation returned exception on [rm1] : org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, so propagating back to caller. 17/05/02 23:53:12 INFO RetryInvocationHandler: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, while invoking $Proxy9.getClusterMetrics over Failover proxy for [rm1, rm2] after 1 failover attempts. Trying to failover after sleeping for 10442ms. [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist Ticket cache: FILE:/tmp/krb5cc_1808 Default principal: ambari-qa@EXAMPLE.COM Valid starting Expires Service principal 05/02/2017 23:53:03 05/03/2017 23:53:03 krbtgt/EXAMPLE.COM@EXAMPLE.COM 05/02/2017 23:53:03 05/03/2017 23:53:03 HTTP/ctr-e133-1493418528701-6489-01-000003.hwx.site@EXAMPLE.COM
Root cause is:
The livy smoke test launched by Ambari is run as livy user, but kinits as ambari-qa, and therefore messes up the credential cache for livy.
Attachments
Attachments
Issue Links
- links to
