[KAFKA-6539] KafkaConsumer endlessly try to connect to a dead broker, ignoring brokers alive

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: consumer
• Labels: windows
• Environment: Java 8 ; Brokers on CentOS 7.4 ; Consumers on Windows 10
• Assignee: Unassigned
• Reporter: Song Younghwan
• Created: 07/Feb/18 03:35
• Updated: 09/Mar/18 00:40

Description

I consider to use Kafka in my company, so currently doing failover test.

Conditions:

• org.apache.kafka:kafka-clients:1.0.0
• New consumer using bootstrap.servers, a consumer group and a group coordinator
• num. brokers = 3 (id #1, #2, #3)
• Topic num. partitions = 3, replication factor = 3
• offsets.topic.replication.factor = 3

Reproduction Step:

1. Run consumers in the same consumer group, each of them subscribe to a topic
2. Kill (kill -9) #1, #2 broker simultaneously (only #3 online)
3. Consumers eventually connect to #3 broker
4. Start #1, #2 broker again after a while (#1, #2, #3 online)
5. Kill (kill -9) #2, #3 broker simultaneously (only #1 online)
6. Now consumers endlessly try to connect to #3 broker only
7. Start #2 broker again after a while (#1, #2 online)
8. Consumers still blindly try to connect to #3 broker

Expectation:

Consumers successfully connect to #1 broker after step 5.

Record:

I attached a consumer log file with TRACE log level. Related events below:

• 12:03:13 kills #1, #2 broker simultaneously
• 12:03:42 starts #1, #2 broker again
• 12:04:01 kills #2, #3 broker simultaneously
• 12:04:42 starts #2 broker again