Minor Description
To avoid triggering auto topic creation (if `auto.create.topic.enable=true` on the brokers), Kafka Streams uses consumer pattern subscription. For this case, the consumer requests all metadata from the brokers and does client side filtering.
However, if users want to set ACL to restrict a Kafka Streams application, this may results in broker side ERROR logs that some metadata cannot be provided. The only way to avoid those broker side ERROR logs is to grant corresponding permissions.
As of 2.3 release it's possible to disable auto topic creation client side (via https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should use this new feature (note, that broker version 0.11 is required) to allow users to set strict ACLs without getting flooded with ERROR logs on the broker.
The proposal is that by default Kafka Streams disables auto-topic create client side (optimistically) and uses regular subscription (not pattern subscription). If an older broker is used, users need to explicitly enable `allow.auto.create.topic` client side. If we detect this setting, we switch back to pattern based subscription.
If users don't enable auto topic create client side and run with an older broker, we would just rethrow the exception to the user, adding some context information on how to fix the issue.
Attachments
Issue Links
- relates to
-
KAFKA-7317 Use collections subscription for main consumer to reduce metadata
-
- Resolved
-