Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11821

Spark-Hive doesn't work with Kerberos in local mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.3
    • Fix Version/s: 1.6.0
    • Component/s: Spark Core
    • Labels:
      None
    • Environment:

      local mode, Kerberos, Hive 1.2.1

      Description

      There is an error between Spark-hive and Hive on a Hadoop cluster. I have found the issue recently fixed: SPARK-10181 , but it doesn't work in local mode.

      I think it's enough to add an "if" to that patch or make the location of a keytab setuped for all environments.

      stacktrace:
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
      at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
      at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
      at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
      at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
      at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
      at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
      at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
      at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:193)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:183)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179)
      at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:228)
      at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:187)
      at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:394)
      at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:176)
      at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:179)
      ...
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
      ... 52 more

        Attachments

          Activity

            People

            • Assignee:
              woj_in Wojciech Indyk
              Reporter:
              woj_in Wojciech Indyk
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: