[HBASE-28321] RpcConnectionRegistry is broken when security is enabled and we use different principal for master and region server

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-2
• Component/s: Client, IPC/RPC, security
• Labels: pull-request-available
• Hadoop Flags: Reviewed

Release Note

Introduced a preamble security call to let server respond its server principal so client knows which one to use if there are multiple server principal candidates.

This requires changing the SecurityInfo to accept a list of server principal patterns for a given rpc service.

Altough SecurityInfo is marked as IA.Private, since it is leaked in SaslClientAuthenticationProvider, we still keep the getServerPrincipal method and only mark it as deprecated since 2.6.0, and plan to remove it in 4.0.0.

In SaslClientAuthenticationProvider, we marked the old createClient method as deprecated since 2.6.0 and plan to remove in 4.0.0. Now you should prefer the method which passes a String server principal, instead of passing a SecurityInfo.

And notice that, if you do use different server principals for master and region server, then if you use 2.6.0+ hbase client, you can not connect a 2.6.0- cluster now, as we do not implement the fallback logic since it is not easy as the server will close the connection directly with unexpected header error. But anyway, you can not connect to the cluster with 2.6.0- client either, because there is no to provide two candidates in the old code base, and this is just what this issue fixes.

• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 18/Jan/24 02:37
• Updated: 29/Feb/24 17:42
• Resolved: 28/Feb/24 15:23

Description

After introducing RpcConnectionRegistry, we let master and region server both implement ClientMetaService.

In our current client architecture, when security is enabled, we rely on the record in SecurityInfo to determine the server principal to use, unfortunately there is only one principal can be specified, so if we use different principal for master and region server, either we can not connect to master, or we can not connect to region server.

And just changing the server principal field in SecurityInfo to an array can not solve the problem, as when connecting, we do not know whether the remote server is a master or region server, so we still can not determine which principal to use...

Anyway, since this has been in our code base since 2.5.0, it is not a new problem, so just set it as critical, not a blocker. But we should find out the solution ASAP.

Issue Links

• is blocked by: HBASE-28377 Fallback to simple is broken for blocking rpc client
• links to: Design Doc
• links to: GitHub Pull Request #5688
• links to: GitHub Pull Request #5707