Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8821

Avoid pattern subscription to allow for stricter ACL settings

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: streams
    • Labels:
      None

      Description

      To avoid triggering auto topic creation (if `auto.create.topic.enable=true` on the brokers), Kafka Streams uses consumer pattern subscription. For this case, the consumer requests all metadata from the brokers and does client side filtering.

      However, if users want to set ACL to restrict a Kafka Streams application, this may results in broker side ERROR logs that some metadata cannot be provided. The only way to avoid those broker side ERROR logs is to grant corresponding permissions.

      As of 2.3 release it's possible to disable auto topic creation client side (via https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should use this new feature (note, that broker version 0.11 is required) to allow users to set strict ACLs without getting flooded with ERROR logs on the broker.

      The proposal is that by default Kafka Streams disables auto-topic create client side (optimistically) and uses regular subscription (not pattern subscription). If an older broker is used, users need to explicitly enable `allow.auto.create.topic` client side. If we detect this setting, we switch back to pattern based subscription.

      If users don't enable auto topic create client side and run with an older broker, we would just rethrow the exception to the user, adding some context information on how to fix the issue. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ableegoldman Sophie Blee-Goldman
                Reporter:
                mjsax Matthias J. Sax
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: