[HBASE-28757] Understand how supportplaintext property works in TLS setup.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: security
• Assignee: Unassigned
• Reporter: Rushabh Shah
• Created: 25/Jul/24 17:41
• Updated: 25/Jul/24 19:24

Description

We are testing TLS feature and I am confused on how hbase.server.netty.tls.supportplaintext property works.
Here is our current setup. This is a fresh cluster deployment.
hbase.server.netty.tls.enabled --> true
hbase.client.netty.tls.enabled --> true
hbase.server.netty.tls.supportplaintext --> false (We don't want to fallback on kerberos)
We still have our kerberos related configuration enabled.
hbase.security.authentication --> kerberos

Our expectation:
During regionserver startup, regionserver will use TLS for authentication and the communication will succeed.

Actual observation
During regionserver startup, hmaster authenticates regionserver* via kerberos authentication*and regionserver's reportForDuty RPC fails.

RS logs:

2024-07-25 16:59:55,098 INFO  [regionserver/regionserver-0:60020] regionserver.HRegionServer - reportForDuty to master=hmaster-0,60000,1721926791062 with isa=regionserver-0/<rs-ip-address>:60020, startcode=1721926793434

2024-07-25 16:59:55,548 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id: 0xa48e3487, L:/<rs-ip-address>:39837 - R:hmaster-0/<hmaster-ip-address>:60000] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

2024-07-25 16:59:55,578 DEBUG [RS-EventLoopGroup-1-2] security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0. (auth:KERBEROS)][action: org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@3769e55]
java.lang.Exception
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
        at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
        at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
		...
		...

2024-07-25 16:59:55,581 DEBUG [RS-EventLoopGroup-1-2] security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0 (auth:KERBEROS)][action: org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@c6f0806]
java.lang.Exception
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
        at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
        at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
        at org.apache.hbase.thirdparty.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)

2024-07-25 16:59:55,602 WARN  [regionserver/regionserver-0:60020] regionserver.HRegionServer - error telling master we are up
org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=hmaster-0:60000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:340)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:595)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:16398)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2997)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.lambda$run$2(HRegionServer.java:1084)
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1079)
Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=hmaster-0:60000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:233)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
        at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
        at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
        at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:231)
        at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:239)
        at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)

Hmaster logs

2024-07-25 16:59:55,378 DEBUG [RS-EventLoopGroup-1-2] ipc.NettyRpcServer - SSL handler added for channel: [id: 0xd4071764, L:/<hmaster-ip>:60000 - R:regionserver-0/<rs-ip>:39837]

2024-07-25 16:59:55,526 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id: 0xd4071764, L:/<hmaster-ip>:60000 - R:regionserver-0/<rs-ip>:39837] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

2024-07-25 16:59:55,583 INFO  [RS-EventLoopGroup-1-2] hbase.Server - Auth successful for hbase/regionserver-0 (auth:KERBEROS)