Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3983

Select on table with hbase storage handler fails with an SASL error

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HBase Handler, Security
    • Labels:
      None
    • Environment:

      hive-0.10
      hbase-0.94.5.5
      hadoop-0.23.3.1
      hcatalog-0.5

      Description

      The table is created using the following query:

      CREATE TABLE hbase_table_1(key int, value string) 
      STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
      WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
      TBLPROPERTIES ("hbase.table.name" = "xyz"); 
      

      Doing a select on the table launches a map-reduce job. But the job fails with the following error:

      2013-02-02 01:31:07,500 FATAL [IPC Server handler 3 on 40118] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1348093718159_1501_m_000000_0 - exited : java.io.IOException: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
      	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
      	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
      	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
      Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:242)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
      	at org.apache.hadoop.hbase.security.User.call(User.java:590)
      	at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
      	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:203)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:291)
      	at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
      	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
      	at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:104)
      	at $Proxy12.getProtocolVersion(Unknown Source)
      	at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:146)
      	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1335)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1291)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:987)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:882)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:984)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:886)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:843)
      	at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
      	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
      	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
      	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
      	... 9 more
      Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
      	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:156)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
      	at org.apache.hadoop.hbase.security.User.call(User.java:590)
      	at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
      	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444)
      	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
      	... 27 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
      	... 44 more
      

        Issue Links

          Activity

          Hide
          amalakar Arup Malakar added a comment -

          I don't have the setup to test this, will reopen if I see it again. Thanks!

          Show
          amalakar Arup Malakar added a comment - I don't have the setup to test this, will reopen if I see it again. Thanks!
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Arup Malakar Like I mentioned, I tested this out on a kerberos enabled cluster and was able to create and run queries just fine. This issue should have been fixed with HIVE-8874 where we added the capability to retrieve the auth tokens from HBase.

          Please feel free to open another one if you continue to see issues.

          Show
          swarnim Swarnim Kulkarni added a comment - Arup Malakar Like I mentioned, I tested this out on a kerberos enabled cluster and was able to create and run queries just fine. This issue should have been fixed with HIVE-8874 where we added the capability to retrieve the auth tokens from HBase. Please feel free to open another one if you continue to see issues.
          Hide
          swarnim Swarnim Kulkarni added a comment -

          Arup Malakar I tested this put on the latest version of hive with kerberos enabled and these queries succeed just fine. Is this still a problem for you or are we good to close this one?

          Show
          swarnim Swarnim Kulkarni added a comment - Arup Malakar I tested this put on the latest version of hive with kerberos enabled and these queries succeed just fine. Is this still a problem for you or are we good to close this one?
          Hide
          ndimiduk Nick Dimiduk added a comment -

          Ping. Does this solution resolve your issue?

          Show
          ndimiduk Nick Dimiduk added a comment - Ping. Does this solution resolve your issue?
          Hide
          alo.alt Alexander Alten-Lorenz added a comment -

          Add the hbase-config into hive-site.xml:

          <property>
          <name>hive.aux.jars.path</name>
          <value>file:///etc/hbase/conf/</value>
          </property>

          Show
          alo.alt Alexander Alten-Lorenz added a comment - Add the hbase-config into hive-site.xml: <property> <name>hive.aux.jars.path</name> <value> file:///etc/hbase/conf/ </value> </property>

            People

            • Assignee:
              swarnim Swarnim Kulkarni
              Reporter:
              amalakar Arup Malakar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development