Description
Currently listTopics would list all the topics in the cluster which often isn't desired due to the large number of topics and because the authorizer would check access for every topic which could result in various bloated access/audit logs that would make them hard to read and also in case of many topics it isn't really effective to return all of them due to the response's size.
To take an example let's have a user who issues a naive command such as:
kafka-topics --list --topic topic6 --bootstrap-server mycompany.com:9092 --command-config client.properties
In this case the admin client would issue a metadata request for all the topics. When this request gets to the broker then it will try to check permissions to every topic in the cluster and then return those which are allowed for "describe" to the client. Then the command when it gets the answer will simply filter out topic6 from the response.
Looking at this problem we can see that if we allow the broker to do the filtering then we can:
- do less permission checks
- return less data to the clients
Regarding the solution I see two path but both of them of course subject community discussion:
- modify the MetadataRequest and put some kind of regex in it
- create a new list API that would contain this filter pattern
Perhaps the list API would make more sense as the MetadataRequest is currently used because it just fits but there is no real reason to extend it with this as the functionality won't be used elsewhere.