[HIVE-3983] Select on table with hbase storage handler fails with an SASL error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: HBase Handler, Security

Environment

hive-0.10
hbase-0.94.5.5
hadoop-0.23.3.1
hcatalog-0.5

• Assignee: Swarnim Kulkarni Assign to me
• Reporter: Arup Malakar
• Created: 04/Feb/13 21:11
• Updated: 14/Aug/15 06:42
• Resolved: 13/Aug/15 05:46

Description

The table is created using the following query:

CREATE TABLE hbase_table_1(key int, value string) 
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "xyz"); 

Doing a select on the table launches a map-reduce job. But the job fails with the following error:

2013-02-02 01:31:07,500 FATAL [IPC Server handler 3 on 40118] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1348093718159_1501_m_000000_0 - exited : java.io.IOException: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:242)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
	at org.apache.hadoop.hbase.security.User.call(User.java:590)
	at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:203)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:291)
	at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
	at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:104)
	at $Proxy12.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:146)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1335)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1291)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1278)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:987)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:882)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:984)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:886)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:843)
	at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getRecordReader(HiveHBaseTableInputFormat.java:92)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
	... 9 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:156)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
	at org.apache.hadoop.hbase.security.User.call(User.java:590)
	at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:444)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
	... 27 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
	... 44 more

Issue Links

• is related to: HIVE-8874 Error Accessing HBase from Hive via Oozie on Kerberos 5.0.1 cluster