[KAFKA-7958] Transactions are broken with kubernetes hosted brokers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1
• Environment: cp-kakfka 2.1.1-1, kafka-streams 2.1.1
• External issue URL: https://stackoverflow.com/questions/54295588/kafka-streams-failed-to-rebalance-error/54779689#54779689
• Assignee: Unassigned
• Reporter: Thomas Dickinson
• Created: 20/Feb/19 06:59
• Updated: 20/Mar/20 17:08
• Resolved: 20/Mar/20 17:08

Description

After a rolling re-start in a kubernetes-like environment, brokers may change IP address.  From our logs it seems that the transaction manager in the brokers never re-resolves the DNS name of other brokers, keeping stale pod IPs.  Thus transactions stop working.  

[2019-02-20 02:20:20,085] WARN [TransactionCoordinator id=1001] Connection to node 0 (khaki-joey-kafka-0.khaki-joey-kafka-headless.hyperspace-dev/10.233.124.181:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

[2019-02-20 02:20:57,205] WARN [TransactionCoordinator id=1001] Connection to node 1 (khaki-joey-kafka-1.khaki-joey-kafka-headless.hyperspace-dev/10.233.122.67:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

This is from the log from broker 1001 which was restarted first, followed by 1 and then 0.  The log entries are from the day after the rolling restart.

I note a similar issue was fixed for clients 2.1.1  https://issues.apache.org/jira/browse/KAFKA-7755.  We are using streams lib 2.1.1

Update We are now testing with Kafka 2.1.1 (docker cp-kafka 5.1.2-1) and it looks like this may be resolved.  Would be good to get confirmation though.

Issue Links

• is related to: KAFKA-7755 Kubernetes - Kafka clients are resolving DNS entries only one time