[ZOOKEEPER-4403] Quorum TLS certificate validation uses wrong name

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.9
• Component/s: quorum, security
• Labels: quorum ssl tls
• Assignee: Unassigned
• Reporter: Marc Richter
• Created: 25/Oct/21 11:33
• Updated: 17/Feb/22 16:48

Description

When using SSL certificates for public IPv6 DNS endpoints as received from some public Service like "Let's Encrypt" for Quorum Encryption, Zookeeper validates the SNAs of that certificate for the IP Address instead of the DNS name, as configured.

As a Result, these certificates can't be used, since no certificates for IPv6 IPs issued.

This has been observed with Zookeeper Version 3.5.9, which is the one bundled in the most recent release of Kafka (2.8.1).

In the affected environment, there is a 3-node-Zookeeper-Cluster, which is configured as this in zookeeper.properties (mind the DNS name!):

server.1=zookeeper1.ourdomain.cloud:2888:3888
server.2=zookeeper2.ourdomain.cloud:2888:3888
server.3=zookeeper3.ourdomain.cloud:2888:3888

All these records do have a public IPv6 entry only.

The SSL certificates from Let's Encrypt are requested and added to the Quorum-Keystores like this:

1. Using https://github.com/acmesh-official/acme.sh
2. Requesting the cert from Let's Encrypt using:

   ./acme.sh --issue --dns dns_nsupdate -d zookeeper1.ourdomain.cloud

   for each system.

3. Merge fullchain- and certificate-file to a single PKCS12 file using:

   openssl pkcs12 -export -in <certfile> -inkey <keyfile> -out <pkcs12_file> -name zookeeper1.ourdomain.cloud \
       -CAfile <fullchainfile> -password <JKS_password>

4. Adding the resulting PKCS12 file to the Quorum Keystore:

   keytool -importkeystore -deststorepass <JKS_password> -destkeypass <JKS_password> -deststoretype pkcs12 \
       -srckeystore <pkcs12_file> -srcstoretype PKCS12 -srcstorepass <JKS_password> -destkeystore <quorum_jks> \
       -alias zookeeper1.ourdomain.cloud

When any of the systems tries to initiate the quorum-connect, their logs state that the remote's Certificates could not be verified, since the SNA-List does not contain the IPv6 address.
For example: This is the log from zookeeper2.ourdomain.cloud when connecting zookeeper3.ourdomain.cloud:

[2021-10-13 15:13:49,960] INFO Received connection request from /2a01:--CUT--:750:47566 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-10-13 15:13:50,094] ERROR Failed to verify host address: 2a01:--CUT--:750 (org.apache.zookeeper.common.ZKTrustManager)
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <2a01:--CUT--:750> doesn't match any of the subject alternative names: [zookeeper3.ourdomain.cloud]

I think the log lines cited clearly show:

1. Zookeeper is picking up the correct certificate from the quorum Keystore, since it states that the request does not match any SNA and lists zookeeper3.ourdomain.cloud only, which it can only know from the certificate itself.
2. Zookeeper is validating the wrong thing here: Even though the config clearly states to use a DNS name, the certificates SNAs alre validated against the IPv6 address that record belongs to instead of the DNS name configured (ERROR Failed to verify host address: 2a01:-CUT-:750)