[KAFKA-1033] Metadata requests do not always return the complete list of available brokers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.8.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Evan Huus
• Created: 29/Aug/13 02:36
• Updated: 23/Apr/15 18:51
• Resolved: 23/Apr/15 18:51

Description

I discovered this while writing a Go client (https://github.com/Shopify/sarama) and it is making one of the issues I'm having rather difficult to solve (https://github.com/Shopify/sarama/issues/15).

In summary: sending a metadata request with an empty list of topics is supposed to return a list of all metadata in the cluster. However, the list of brokers is incomplete. I have not been able to pin down precisely which brokers are missing, but I believe it happens when a broker is not currently the leader for any partition of any topic.

Among other things this can make it very difficult to provide failover in a small cluster of only one master and one replica server - clients requesting metadata sometimes are not told of the replica broker and cannot fail-over to it when the master goes down.

If it is intentional to only return a subset of brokers (whatever that subset is), please document somewhere what that subset is, and how clients should learn of brokers outside that subset.

Thanks,
Evan