[HBASE-25987] Make SSL keystore type configurable for HBase ThriftServer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.4
• Fix Version/s: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
• Component/s: Thrift
• Assignee: Mate Szalay-Beko
• Reporter: Mate Szalay-Beko
• Created: 09/Jun/21 08:24
• Updated: 10/Jun/21 13:48
• Resolved: 09/Jun/21 16:02

Description

We get the following exception, when trying to start Hbase Thrift Server in http mode (hbase.regionserver.thrift.http=true) and use non default (not "jks") keystore type:

 

2021-06-08 07:40:10,275 ERROR org.apache.hadoop.hbase.thrift.ThriftServer: Cannot run ThriftServer
java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
        at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1197)
        at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:321)
        at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.hadoop.hbase.thrift.ThriftServer$2.run(ThriftServer.java:861)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.hbase.thrift.ThriftServer.run(ThriftServer.java:855)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:882)

This problem appeared after we applied HBASE-25930 to our local HBase version. It looks, we never had a parameter to specify the keystore type for thrift http server. Before HBASE-25930, the keystore type used by the thrift http server was accidentally defined based on the InfoServer (web ui) configuration of "ssl.server.keystore.type". Before HBASE-25930, the InfoServer was started first and it set the keystore type in the global keystore manager, which setting propagated to the thrift http server too, without any override. In HBASE-25930 the startup order changed, and the thrift http server configuration happens before the InfoServer start, so we lack this accidental configuration change now.

Given that we have independent keystore file path / password parameters already for the thrift http server, the proper solution is to create a new parameter also for the keystore type of the thrift http server: hbase.thrift.ssl.keystore.type (defaulting to "jks").

Issue Links

• is caused by: HBASE-25930 Thrift does not support requests in Kerberos environment
• links to: GitHub Pull Request #3367