[HBASE-27204] BlockingRpcClient will hang for 20 seconds when SASL is enabled after finishing negotiation

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.5.0, 3.0.0-alpha-4, 2.4.14
• Component/s: rpc, sasl, security
• Hadoop Flags: Reviewed

Release Note

When Kerberos authentication succeeds, on the server side, after receiving the final SASL token from the client, we simply wait for the client to continue by sending the connection header. After HBASE-24579, on the client side, an additional readStatus() was added, which assumed that after negotiation has completed a status code will be sent. However when authentication has succeeded the server will not send one. As a result the client would hang and only throw an exception when the configured read timeout is reached, which is 20 seconds by default. This was especially noticeable when using BlockingRpcClient as the client implementation. HBASE-24579 was reverted to correct this issue.

• Assignee: Andrew Kyle Purtell
• Reporter: Duo Zhang
• Created: 14/Jul/22 13:43
• Updated: 26/Jul/22 05:37
• Resolved: 25/Jul/22 06:15

Description

Found this when implementing HBASE-27185. When running TestSecureIPC, if BlockingRpcClient is used, the tests will spend much more time comparing to NettyRpcClient.

The problem is that, for the normal kerberos authentication, the last step is client send a reply to server, so after server receives the last token, it will not write anything back but expect client to send connection header.

In HBASE-24579, for reading the error message, we added a readReply after the SaslClient indicates that the negotiation is completed. But as said above, for normal cases, we will not write anything back from server side, so the client will hang there and only throw an exception when timeout is reached, which is 20 seconds.

This nearly makes the BlockingRpcClient unusable when sasl is enabled, as it will hang 20 seconds when connecting...

Issue Links

• is broken by: HBASE-24579 Failed SASL authentication does not result in an exception on client side
• links to: GitHub Pull Request #4642