[KAFKA-7229] Failed to dynamically update kafka certificate in kafka 2.0.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.0.0
• Component/s: security
• Environment: Ubuntu 14.04.5 LTS
• Assignee: Unassigned
• Reporter: Yu Yang
• Created: 01/Aug/18 16:42
• Updated: 01/Aug/18 17:05
• Resolved: 01/Aug/18 16:54

Description

In kafka 1.1, we use the following command in a cron job to dynamically update the certificate that kafka uses :

kafka-configs.sh --bootstrap-server localhost:9093 --command-config /var/pinterest/kafka/client.properties --alter --add-config listener.name.ssl.ssl.keystore.location=/var/certs/kafka/kafka.keystore.jks.1533141082.38 --entity-type brokers --entity-name 9

In kafka 2.0.0, the command fails with the following exception:

[2018-08-01 16:38:01,480] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Error while executing config command with args '--bootstrap-server localhost:9093 --command-config /var/pinterest/kafka/client.properties --alter --add-config listener.name.ssl.ssl.keystore.location=/var/pinterest/kafka/kafka.keystore.jks.1533141082.38 --entity-type brokers --entity-name 9'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
at kafka.admin.ConfigCommand$.brokerConfig(ConfigCommand.scala:346)
at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:304)
at kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:290)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:83)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
... 7 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching localhost found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:204)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:95)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)