[KYLIN-4884] spark cube in kerberos env

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: v3.1.0
• Component/s: Storage - HBase

Environment

OS ---> Centos 7.6
kerberos--> krb5-server-1.15.1-18
apache-kylin-3.1.0-bin-cdh60
CDH 6.3.3
HDFS 3.0.0+cdh6.3.3
YARN 3.0.0+cdh6.3.3
HBase 2.1.0+cdh6.3.3
Hive 2.1.1+cdh6.3.3
spark 2.4.0+cdh6.3.3
ZooKeeper 3.4.5+cdh6.3.3

• Skill Level: Don't Know (Unsure) - The default level
• Assignee: Unassigned
• Reporter: feiren kuang
• Created: 23/Jan/21 13:33
• Updated: 23/Jan/21 13:33

Description

使用spark engine

kylin start 之前 使用kinit 做好了kerberos认证 ，使用hdfs/hbase cli 访问都正常

在 8 Step Name: Convert Cuboid Data to HFile 

spark log中大量 提示Kerberos认证失败；但是最终任务是成功的

 

21/01/23 20:43:53 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh67.test.com/192.168.1.67:2181, sessionid = 0x2768daceed1189a, negotiated timeout = 120000
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=3, exceptions:
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller{globalStartTime=1611405835836, pause=100, maxAttempts=3}, javax.security.sasl.SaslException: Call to cdh67.test.com/192.168.1.67:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]]
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller{globalStartTime=1611405835836, pause=100, maxAttempts=3}, java.io.IOException: Call to cdh67.test.com/192.168.1.67:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller{globalStartTime=1611405835836, pause=100, maxAttempts=3}, java.io.IOException: Call to cdh67.test.com/192.168.1.67:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.

at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Call to cdh67.test.com/192.168.1.67:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:221)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:42722)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
... 4 more
Caused by: java.io.IOException: Can not send request because relogin is in progress.
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421)
... 16 more