[KAFKA-2738] Can't set SSL as inter-broker-protocol by rolling restart of brokers

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.9.0.0
• Assignee: Ben Stopford
• Reporter: Gwen Shapira
• Reviewer: Jun Rao
• Created: 04/Nov/15 00:48
• Updated: 14/Mar/24 12:05
• Resolved: 05/Nov/15 17:05

Description

Scenario (as carefully documented by benstopford:

1. Start 2 or more brokers with listeners on both PLAINTEXT and SSL protocols, and PLAINTEXT as security.inter.broker.protocol:

inter.broker.protocol.version = 0.9.0.X
security.inter.broker.protocol = PLAINTEXT
listeners = PLAINTEXT://:9092,SSL://:9093

2. Stop one of the brokers and change security.inter.broker.protocol to SSL

inter.broker.protocol.version = 0.9.0.X
security.inter.broker.protocol = SSL
listeners = PLAINTEXT://:9092,SSL://:9093

3. Start that broker again.

You will get replication errors as it will attempt to use SSL on a PLAINTEXT port:

WARN ReplicaFetcherThread-0-3, Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@78ca3ba1. Possible cause: java.io.IOException: Connection to Node(3, worker4, 9092) failed (kafka.server.ReplicaFetcherThread)
WARN Failed to send SSL Close message (org.apache.kafka.common.network.SslTransportLayer)
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:188)
at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:161)
at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:50)
at org.apache.kafka.common.network.Selector.close(Selector.java:448)
at org.apache.kafka.common.network.Selector.poll(Selector.java:316)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
at kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:128)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:105)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:58)
at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:202)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:192)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:102)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

Issue Links

• links to: GitHub Pull Request #428