Well I think there are actually two problems.
The first is that it is mixing concerns in kind of a messy way. In some sense it would be the "get metadata or maybe create a new topic" api. So I would rather not enshrine that in the public API. I think it is okay the kind of hacky way it is now, and in the future we can add an api.
The second problem is that we need to move the per-topic config into zookeeper so that you can dynamically add topics with their own settings (flush interval, retention period, # of partitions, etc...there are like ten of these settings). This is discussed in
KAFKA-554. Currently these are in the broker config, but that means bouncing the broker all the time which is a hassle. So we will need to include these in the api that creates topics. That makes it messier still since we would have all these properties mixed into the get_metadata call.
I agree that right now it is a lot of overhead to add an api, but I think we could fix that directly by making it easier to add apis.
KAFKA-643 had one proposal for htis.
Since you just went through that it would be good to get your feedback on that proposal.