[HBASE-25338] Client app getting "Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane." errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Client

Environment

Docker container build on adoptopenjdk/openjdk14:jre or adoptopenjdk/openjdk15:jre

Spring Boot 2.0.1

hbase-client 2.3.0

hbase-server 2.3.0

 

• Assignee: Unassigned
• Reporter: ALEXEY POLYAKOV
• Created: 29/Nov/20 00:16
• Updated: 29/Nov/20 00:53
• Resolved: 29/Nov/20 00:36

Description

Application is ran from Spring jar, hbase-client jar is included.

But looks like ServiceLoader.load is unable to load providers for SaslClientAuthenticationProvider

I get lot of stack traces like this in log file:

2020-11-29T00:08:37.019Z, RpcRetryingCaller{globalStartTime=1606608514899, pause=100, maxAttempts=6}, java.io.IOException: Call to hostname/xx.xx.xx.xx:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Call to hostname/xx.xx.xx.xx:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at java.base/jdk.internal.reflect.GeneratedConstructorAccessor101.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:225)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:378)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:89)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
	at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
	at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:132)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:316)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:89)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:572)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:45390)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
	at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
	at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
	at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:396)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:370)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
	... 4 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:159)
	... 17 more
Caused by: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at org.apache.hadoop.hbase.security.provider.BuiltInProviderSelector.configure(BuiltInProviderSelector.java:97)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiateSelector(SaslClientAuthenticationProviders.java:116)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiate(SaslClientAuthenticationProviders.java:190)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.getInstance(SaslClientAuthenticationProviders.java:78)
	at org.apache.hadoop.hbase.ipc.RpcConnection.<init>(RpcConnection.java:103)
	at org.apache.hadoop.hbase.ipc.NettyRpcConnection.<init>(NettyRpcConnection.java:97)
	at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:76)
	at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:39)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:350)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:419)
	... 16 more

 I got this problem after changing client version from 2.1.9 to 2.3.0 (I've upgraded server to 2.3.0)

When I switch back to 2.1.9 client, it works without errors.