[KAFKA-3068] NetworkClient may connect to a different Kafka cluster than originally configured

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0.0
• Fix Version/s: 0.9.0.1, 0.10.0.0
• Component/s: clients
• Assignee: Eno Thereska
• Reporter: Jun Rao
• Reviewer: Ewen Cheslack-Postava
• Created: 06/Jan/16 00:57
• Updated: 18/Jan/23 11:52
• Resolved: 02/Feb/16 17:36

Description

In https://github.com/apache/kafka/pull/290, we added the logic to cache all brokers (id and ip) that the client has ever seen. If we can't find an available broker from the current Metadata, we will pick a broker that we have ever seen (in NetworkClient.leastLoadedNode()).

One potential problem this logic can introduce is the following. Suppose that we have a broker with id 1 in a Kafka cluster. A producer client remembers this broker in nodesEverSeen. At some point, we bring down this broker and use the host in a different Kafka cluster. Then, the producer client uses this broker from nodesEverSeen to refresh metadata. It will find the metadata in a different Kafka cluster and start producing data there.

Issue Links

• duplicates: KAFKA-13653 Proactively discover alive brokers from bootstrap server lists when all nodes are down
• is related to: KAFKA-14548 Stable streams applications stall due to infrequent restoreConsumer polls
• relates to: KAFKA-12480 Reuse bootstrap servers in clients when last alive broker in cluster metadata is unavailable
• relates to: KAFKA-13467 Clients never refresh cached bootstrap IPs
• mentioned in: Page Loading...