[HBASE-28106] TestShadeSaslAuthenticationProvider fails for branch-2.x

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.6.0, 2.4.18, 2.5.6
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Nihal Jain
• Created: 21/Sep/23 20:30
• Updated: 25/Sep/23 19:42
• Resolved: 24/Sep/23 13:51

Description

org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider is failing consistently in branch-2.x

This could be a code bug as well!

 

For branch-2:

• Nightly failing: https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/885/testReport/

For branch-2.5

• PR failing: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5422/3/testReport/
• Nightly failing: https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/405/testReport/

For branch-2.4

• PR failing:[ https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5423/3/testReport/
• Nightly failing: https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/619/testReport/

 

StackTrace copied from failed test case:

java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.hbase.DoNotRetryIOException> but was<org.apache.hadoop.hbase.client.RetriesExhaustedException>
	at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:30)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:07.490Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:06.869Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=6a87b11f0f42:38383 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
2023-09-21T17:53:06.975Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.182Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.489Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:06.862Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer

2023-09-21T17:53:08.215Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:07.597Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.703Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:07.909Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:08.215Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:07.591Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer

2023-09-21T17:53:09.057Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:08.422Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:08.530Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=6a87b11f0f42:38383 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
2023-09-21T17:53:08.748Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.056Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:08.416Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=6a87b11f0f42:38383 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed

2023-09-21T17:53:09.990Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:52:04.943Z, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:09.362Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.469Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.683Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.990Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer


	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
	at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
	at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
	at org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:235)
	at org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider$3.run(TestShadeSaslAuthenticationProvider.java:230)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.apache.hadoop.hbase.security.provider.example.TestShadeSaslAuthenticationProvider.testNegativeAuthentication(TestShadeSaslAuthenticationProvider.java:230)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
	... 18 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
2023-09-21T17:53:09.362Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.469Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.683Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
2023-09-21T17:53:09.990Z, RpcRetryingCaller{globalStartTime=2023-09-21T17:53:09.357Z, pause=100, maxAttempts=4}, java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer

	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:141)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:390)
	at org.apache.hadoop.hbase.client.HTable.lambda$get$0(HTable.java:363)
	at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:363)
	at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1141)
	at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235)
	at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:742)
	at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:214)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
	... 37 more
Caused by: java.io.IOException: Call to address=6a87b11f0f42:38383 failed on local exception: java.io.IOException: Connection reset by peer
	at java.base/jdk.internal.reflect.GeneratedConstructorAccessor69.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:239)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
	at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
	at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:212)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.exceptionCaught(NettyRpcDuplexHandler.java:229)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more
Caused by: java.io.IOException: Connection reset by peer
	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
	at org.apache.hbase.thirdparty.io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259)
	at org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
	at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
	... 8 more

 
Also note the test class code is not same between master and branch-2.x

• https://github.com/apache/hbase/blob/master/hbase-examples/src/test/java/org/apache/hadoop/hbase/security/provider/example/TestShadeSaslAuthenticationProvider.java
• https://github.com/apache/hbase/blob/branch-2/hbase-examples/src/test/java/org/apache/hadoop/hbase/security/provider/example/TestShadeSaslAuthenticationProvider.java 

This issue could be related to HBASE-28101

Issue Links

• is related to: HBASE-28101 Should check the return value of protobuf Message.mergeDelimitedFrom
• relates to: HBASE-28110 Align TestShadeSaslAuthenticationProvider between different branches
• links to: GitHub Pull Request #5433