[KAFKA-5015] SASL/SCRAM authentication failures are hidden

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.2.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Johan Ström
• Created: 05/Apr/17 13:43
• Updated: 22/Sep/17 11:36
• Resolved: 22/Sep/17 11:36

Description

During experimentation with multiple brokers and SCRAM authentication, the brokers didn't seem to connect properly.
Apparently the receiving server does not log connection failures (and their cause) unless you enable DEBUG logging on org.apache.kafka.common.network.Selector.

Expected: that the rejected connections is logged (without stack trace) without having to enable DEBUG.

(The root cause of my problem was that I hadn't yet added the user to the Zk-backed SCRAM configuration)

The controller flooded controller.log with WARNs:

[2017-04-05 15:33:42,850] WARN [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker kafka02:9093 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to kafka02:9093 (id: 1 rack: null) failed

The peer does not log anything in any log, until debugging was enabled:

[2017-04-05 15:28:58,373] DEBUG Accepted connection from /10.10.0.5:43670 on /10.10.0.6:9093 and assigned it to processor 4, sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400] (kafka.network.Acceptor)
[2017-04-05 15:28:58,374] DEBUG Processor 4 listening to new connection from /10.10.0.5:43670 (kafka.network.Processor)
[2017-04-05 15:28:58,376] DEBUG Set SASL server state to HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-04-05 15:28:58,376] DEBUG Handle Kafka request SASL_HANDSHAKE (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-04-05 15:28:58,378] DEBUG Using SASL mechanism 'SCRAM-SHA-512' provided by client (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-04-05 15:28:58,381] DEBUG Setting SASL/SCRAM_SHA_512 server state to RECEIVE_CLIENT_FIRST_MESSAGE (org.apache.kafka.common.security.scram.ScramSaslServer)
[2017-04-05 15:28:58,381] DEBUG Set SASL server state to AUTHENTICATE (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-04-05 15:28:58,383] DEBUG Setting SASL/SCRAM_SHA_512 server state to FAILED (org.apache.kafka.common.security.scram.ScramSaslServer)
[2017-04-05 15:28:58,383] DEBUG Set SASL server state to FAILED (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-04-05 15:28:58,385] DEBUG Connection with /10.10.0.5 disconnected (org.apache.kafka.common.network.Selector)
java.io.IOException: javax.security.sasl.SaslException: Authentication failed: Credentials could not be obtained [Caused by javax.security.sasl.SaslException: Authentication failed: Invalid user credentials]
	at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:250)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:71)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:350)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
	at kafka.network.Processor.poll(SocketServer.scala:494)
	at kafka.network.Processor.run(SocketServer.scala:432)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.security.sasl.SaslException: Authentication failed: Credentials could not be obtained [Caused by javax.security.sasl.SaslException: Authentication failed: Invalid user credentials]
	at org.apache.kafka.common.security.scram.ScramSaslServer.evaluateResponse(ScramSaslServer.java:104)
	at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:235)
	... 6 more
Caused by: javax.security.sasl.SaslException: Authentication failed: Invalid user credentials
	at org.apache.kafka.common.security.scram.ScramSaslServer.evaluateResponse(ScramSaslServer.java:94)
	... 7 more

Issue Links

• is duplicated by: KAFKA-4764 Improve diagnostics for SASL authentication failures