[ZOOKEEPER-3882] Zookeeper quorum formation fails when TLS is enabled in k8s env

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.5.7
• Component/s: leaderElection, quorum
• Assignee: Unassigned
• Reporter: Bharath B
• Created: 07/Jul/20 20:40
• Updated: 14/Jul/20 11:41

Description

Have deployed zookeeper(v3.5.7) included in the kafka(v2.5.0) bundle as containers using kubernetes, where 3 instances of each kafka and zookeeper is deployed. Interaction among kafka brokers, with kafka client and kafka with zookeeper are all TLS based and is working as expected. But zookeeper quorum formation fails with TLS handshake error, as the server name in the https request does not match with any of the SANs in the certificate configured for zookeeper server. Server name in the request is of the form "x-x-x-x.kubernetes.default.svc.cluster.local" (where x-x-x-x is the IP address of the POD), and I am unable to understand the reason behind pre-pending FQDN with a IP address. Could anyone please let me know, if I am missing any configuration or the behavior observed is as designed. Like in kafka, we don't have "ssl.client.auth" confiuration parameter for zookeeper, so I am not so sure, if it's client or server validation failing during the handshake.

Please find below the extract of the error logs from the zookeeper1 POD

ERROR Failed to verify host address: x.x.x.x (org.apache.zookeeper.common.ZKTrustManager)
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <x.x.x.x> doesn't match any of the subject alternative names: [zookeeper, zookeeper1, zookeeper2, zookeeper3, zookeeper1.default.svc.cluster.local, zookeeper2.default.svc.cluster.local, zookeeper3.default.svc.cluster.local]