[HBASE-12953] RegionServer is not functionally working with AysncRpcClient in secure mode

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.1.0, 2.0.0
• Fix Version/s: 1.1.0, 2.0.0
• Component/s: security
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Ashish Singhi
• Created: 02/Feb/15 09:43
• Updated: 03/Jun/15 15:29
• Resolved: 24/Feb/15 04:45

Description

HBase version 2.0.0
Default value for hbase.rpc.client.impl is set to AsyncRpcClient.
When trying to install HBase with Kerberos, RegionServer is not working functionally.
The following log is logged in its log file

2015-02-02 14:59:05,407 WARN  [AsyncRpcChannel-pool1-t1] channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.ChannelPipelineException: org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded() has thrown an exception; removed.
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:499)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded(DefaultChannelPipeline.java:481)
	at io.netty.channel.DefaultChannelPipeline.addFirst0(DefaultChannelPipeline.java:114)
	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:97)
	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:235)
	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:214)
	at org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:194)
	at org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:157)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:253)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:288)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded(SaslClientHandler.java:154)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:486)
	... 20 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

When set hbase.rpc.client.impl to RpcClientImpl, there seems to be no issue.

Issue Links

• links to: reviewboard