[HBASE-27027] Deprecated jetty SslContextFactory cause HMaster startup failure due to multiple certificates in KeyStores

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.11
• Fix Version/s: 2.5.0, 3.0.0-alpha-3, 2.4.13
• Component/s: security
• Hadoop Flags: Reviewed
• Assignee: Shinya Yoshida
• Reporter: Shinya Yoshida
• Created: 12/May/22 06:31
• Updated: 23/May/22 05:39
• Resolved: 22/May/22 07:01

Description

When we start hbase 2.4.11 masters with secure configuration (using https), we got this exception and failed to start the cluster

2022-05-12 15:06:05,447 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2951)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:253)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:149)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:152)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2962)
Caused by: java.io.IOException: Problem starting http server
        at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:1140)
        at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:151)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:2230)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:689)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:419)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2944)
        ... 5 more
Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1288)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1270)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:372)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:243)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.apache.hbase.thirdparty.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:1109)
        ... 15 more

Some knowledge base said using SslContextFactory.Server solves this issue as the exception message said
https://kb.vmware.com/s/article/83778

Actually, officially SslContextFactory base constructor deprecated and use either of Server or Client is recommended
https://github.com/eclipse/jetty.project/blob/8da83308eeca865e495e53ef315a249d63ba9332/jetty-util/src/main/java/org/eclipse/jetty/util/ssl/SslContextFactory.java#L188-L233

Ref.

• https://github.com/eclipse/jetty.project/issues/3464
• https://github.com/eclipse/jetty.project/pull/4386

Issue Links

• relates to: NIFI-7730 Jetty server does not start up when a keystore with multiple certificates is used
• links to: GitHub Pull Request #4425