[KAFKA-13146] Consider client use cases for accessing controller endpoints

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: kip-500
• Assignee: Unassigned
• Reporter: Jason Gustafson
• Created: 28/Jul/21 16:58
• Updated: 29/Jul/21 13:31

Description

In KAFKA-13143, we dropped the Metadata from the controller APIs. We did this for two reasons. First, the implementation did not return any topic metadata. This was confusing for users who mistakenly tried to use the controller endpoint in order to describe or list topics since it would appear that no topics existed in the cluster. The second reason is that the implementation returned the controller endpoints. So even if we returned the topic metadata, clients would be unable to access the topics for reading or writing through the controller endpoint.

So for 3.0, we are effectively saying that clients should only access the broker endpoints. Long term, is that what we want? When running the controllers as separate nodes, it may be useful to initialize the controllers and cluster metadata before starting any of the brokers, for example. For this to work, we need to put some thought into how the Metadata API should work with controllers. For example, we can return a flag or some kind of error code in the response to indicate that topic metadata is not available. We have also considered whether the internal __cluster_metadata topic should be readable through the controller endpoints by consumers.