[KAFKA-13372] failed authentication due to: SSL handshake failed - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.2.2
Fix Version/s: None
Component/s: clients
Labels:
None

Description

Hi everyone,

I have the next issue about authentication SCRAM + SSL. I’m using the CLI and this is the version of my client (./kafka_2.13-2.8.1/bin/kafka-topics.sh). In this example I will talk about list topics, but another operations (consumer, producer) failed too.

First, let me describe the current scenario:

I have 5 Kafka servers with
kafka-broker-0.mydomain.com
kafka-broker-1.mydomain.com
kafka-broker-2.mydomain.com
kafka-broker-3.mydomain.com
kafka-broker-4.mydomain.com

I have a DNS principal configured with Round Robin to IPs broker:
kafka-broker-princial.mydomain.com (Round Robin)

I have configured for each broker the next listeners (I'm using 3 ports):

advertised.listeners=SASL_SSL://kafka-broker-0.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-0.mydomain.com:9093,PLAINTEXT://kafka-broker-0.mydomain.com:9092

9092 for PLAINTEXT
9093 for SASL_PLAINTEXT
9094 for SASL_SSL

My Kafka broker servers have the next config server.properties:

advertised.listeners=SASL_SSL://kafka-broker-X.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-X.mydomain.com:9093,PLAINTEXT://kafka-broker-X.mydomain.com:9092
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
auto.create.topics.enable=false
auto.leader.rebalance.enable=true
background.threads=10
broker.id=X
broker.rack=us-east-1c
compression.type=producer
connections.max.idle.ms=2700000
controlled.shutdown.enable=true
delete.topic.enable=true
host.name=localhost
leader.imbalance.check.interval.seconds=300
leader.imbalance.per.broker.percentage=10
listeners=SASL_SSL://0.0.0.0:9094,SASL_PLAINTEXT://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
log.cleaner.enable=true
log.dirs=/var/lib/kafka/log/data1,/var/lib/kafka/log/data2,/var/lib/kafka/log/data3
log.retention.check.interval.ms=300000
log.retention.hours=336
log.segment.bytes=1073741824
message.max.bytes=1000012
min.insync.replicas=2
num.io.threads=8
num.network.threads=3
num.partitions=3
num.recovery.threads.per.data.dir=1
num.replica.fetchers=1
offset.metadata.max.bytes=4096
offsets.commit.timeout.ms=5000
offsets.retention.minutes=129600
offsets.topic.num.partitions=50
offsets.topic.replication.factor=3
port=9092
queued.max.requests=500
replica.fetch.min.bytes=1
replica.fetch.wait.max.ms=500
sasl.enabled.mechanisms=SCRAM-SHA-256,GSSAPI
sasl.kerberos.service.name=xxxxx
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
security.inter.broker.protocol=SASL_SSL
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
ssl.client.auth=required
ssl.endpoint.identification.algorithm=""
ssl.enabled.protocols=TLSv1.2
ssl.key.password=xxxx
ssl.keystore.location=/etc/ssl/default_keystore.jks
ssl.keystore.password=xxxxxxxx
ssl.truststore.location=/usr/lib/jvm/java-11-adoptopenjdk-hotspot/lib/security/cacerts
ssl.truststore.password= xxxxxxxx
ssl.truststore.type=JKS
super.users=User:xxxxx
zookeeper.connect=kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com :2181,kafka-zk-X.mydomain.com:218/my-environment
zookeeper.connection.timeout.ms=6000
zookeeper.sasl.client=false

I was trying the next things:

PLAINTEXT: I can consume directly to broker to broker with port 9092 (Using IP or dns broker)
PLAINTEXT: I also can consume directly to DNS principal configured with Round Robin with port 9092 (Using DNS principal)
SASL_SSL: I can consume directly to broker to broker with port 9094 (Using only dns broker due it needs to validate the certificate)
SASL_SSL: I cannot consume directly to DNS principal configured with Round Robin with port 9094

The issue is: * (x)SASL_SSL: I cannot consume directly to DNS principal configured with Round Robin with port 9094. Only I have the issue with I try to connect directly to DNS principal. My certificates contains permissions with all my subdomains under the domain.

I have the next file.config when that I use when I try to connect to DNS principal. (Is the same file that I used for consume directly to broker to broker with port 9094)

Required connection configs for Kafka producer, consumer, and admin

ssl.keystore.location=/My/Path/default_keystore.jks
ssl.keystore.password=xxxxx
ssl.truststore.location=/My/Path/cacerts
ssl.truststore.password= xxxxx
ssl.truststore.type=JKS
ssl.enabled.protocols=TLSv1.2
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=‘xxxxx' password=‘xxxxxx';
client.dns.lookup=use_all_dns_ips

The command that I'm using to try consume directly principal kafka DNS:

$ ./kafka_2.13-2.8.1/bin/kafka-topics.sh --bootstrap-server kafka-broker-princial.mydomain.com:9094 --command-config java9094.config --list
[2021-10-13 01:04:58,206] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-broker-princial.mydomain.com/10.110.209.136:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-10-13 01:04:58,207] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found.
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:412)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
... 19 more
Error while executing topic command : SSL handshake failed
[2021-10-13 01:04:58,212] ERROR org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found.
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:412)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
... 19 more
(kafka.admin.TopicCommand$)

Can you help me with this issue?

Thanks for reading me!

@maisfloro

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Maria Isabel Florez Rodriguez

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 13/Oct/21 04:23

Updated:: 24/Feb/23 20:01

Resolved:: 24/Feb/23 20:01