[HIVE-27675] Support keystore/truststore types for hive to zookeeper integration points

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 4.0.0
• Component/s: HiveServer2, JDBC, Standalone Metastore
• Labels: pull-request-available
• Assignee: Naveen Gangam
• Reporter: Naveen Gangam
• Created: 08/Sep/23 17:16
• Updated: 29/Mar/24 17:49
• Resolved: 15/Sep/23 14:39

Description

In HIVE-24253, we added support for HS2/HMS/JDBC DRiver to support other store types like BCFKS (other than JKS). This allows JDBC Clients to connect to HS2 directly. However, with service discovery enabled, the clients have to connect zookeeper to determine HS2 endpoints. This connectivity currently does not support other store types. Similarly, HS2/HMS services also do not provide ability to use different store types for the zk registration process.

$ beeline 
Connecting to jdbc:hive2://<snip>:2181/default;httpPath=cliservice;principal=hive/_HOST@<SNIP>;retries=5;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;transportMode=http;trustStorePassword=RoeCFK11Pq54;trustStoreType=bcfks;zooKeeperNamespace=hiveserver2
Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0) 

Opening socket connection to server <SNIP>:2182. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
2023-08-09 13:28:07,591 WARN  io.netty.channel.ChannelInitializer: [nioEventLoopGroup-3-1]: Failed to initialize a channel. Closing: [id: 0x0937583f]
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create KeyManager
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:346) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.common.X509Util.createSSLContext(X509Util.java:278) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initSSL(ClientCnxnSocketNetty.java:454) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:444) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.ClientCnxnSocketNetty$ZKClientPipelineFactory.initChannel(ClientCnxnSocketNetty.java:429) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) [netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_382]
Caused by: org.apache.zookeeper.common.X509Exception$KeyManagerException: java.io.IOException: Invalid keystore format
        at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:471) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:344) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        ... 23 more
Caused by: java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:666) ~[?:1.8.0_382]
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57) ~[?:1.8.0_382]
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_382]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71) ~[?:1.8.0_382]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_382]
        at org.apache.zookeeper.common.StandardTypeFileKeyStoreLoader.loadKeyStore(StandardTypeFileKeyStoreLoader.java:54) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.common.X509Util.loadKeyStore(X509Util.java:400) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.common.X509Util.createKeyManager(X509Util.java:460) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:344) ~[zookeeper-3.5.5.7.2.16.300-7.jar:3.5.5.7.2.16.300-7]
        ... 23 more
2023-08-09 13:28:07,591 INFO  org.apache.zookeeper.ClientCnxnSocketNetty: [nioEventLoopGroup-3-1]: future isn't success, cause:
io.netty.channel.StacklessClosedChannelException: null
        at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]

Issue Links

• depends upon: HIVE-24253 HMS and HS2 needs to support keystore/truststores types besides JKS by config
• links to: GitHub Pull Request #4691