Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3396

Unauthorized topics are returned to the user

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0, 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: security
    • Labels:
      None

      Description

      Kafka's clients and protocol exposes unauthorized topics to the end user. This is often considered a security hole. To some, the topic name is considered sensitive information. Those that do not consider the name sensitive, still consider it more information that allows a user to try and circumvent security. Instead, if a user does not have access to the topic, the servers should act as if the topic does not exist.

      To solve this some of the changes could include:

      • The broker should not return a TOPIC_AUTHORIZATION(29) error for requests (metadata, produce, fetch, etc) that include a topic that the user does not have DESCRIBE access to.
      • A user should not receive a TopicAuthorizationException when they do not have DESCRIBE access to a topic or the cluster.
      • The client should not maintain and expose a list of unauthorized topics in org.apache.kafka.common.Cluster.

      Other changes may be required that are not listed here. Further analysis is needed.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1946

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1946
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hachikuji opened a pull request:

          https://github.com/apache/kafka/pull/1946

          MINOR: Follow-up minor improvements/cleanup for KAFKA-3396

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hachikuji/kafka followup-for-kafka-3396

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1946.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1946


          commit efd843430651aacc90d3d2b4c7f3d635416c24be
          Author: Jason Gustafson <jason@confluent.io>
          Date: 2016-10-01T05:42:22Z

          MINOR: Follow-up minor improvements/cleanup for KAFKA-3396


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/1946 MINOR: Follow-up minor improvements/cleanup for KAFKA-3396 You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka followup-for-kafka-3396 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1946.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1946 commit efd843430651aacc90d3d2b4c7f3d635416c24be Author: Jason Gustafson <jason@confluent.io> Date: 2016-10-01T05:42:22Z MINOR: Follow-up minor improvements/cleanup for KAFKA-3396
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1908

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1908
          Hide
          hachikuji Jason Gustafson added a comment -

          Issue resolved by pull request 1908
          https://github.com/apache/kafka/pull/1908

          Show
          hachikuji Jason Gustafson added a comment - Issue resolved by pull request 1908 https://github.com/apache/kafka/pull/1908
          Hide
          ecomar Edoardo Comar added a comment -
          Show
          ecomar Edoardo Comar added a comment - new PR is https://github.com/apache/kafka/pull/1908
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user edoardocomar closed the pull request at:

          https://github.com/apache/kafka/pull/1428

          Show
          githubbot ASF GitHub Bot added a comment - Github user edoardocomar closed the pull request at: https://github.com/apache/kafka/pull/1428
          Hide
          ijuma Ismael Juma added a comment -

          Marking this as "Critical" so that we can make sure we get to it for 0.10.1.0.

          Show
          ijuma Ismael Juma added a comment - Marking this as "Critical" so that we can make sure we get to it for 0.10.1.0.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user edoardocomar opened a pull request:

          https://github.com/apache/kafka/pull/1428

          KAFKA-3396 : Unauthorized topics are returned to the user

          Modified KafkaApis to return Errors.UNKNOWN_TOPIC_OR_PARTITION if
          principal has no Describe access to topic

          Unit tests expanded

          Some paths cause the client to block due to bug
          https://issues.apache.org/jira/browse/KAFKA-3727?filter=-2
          tests work around this by executing in separate thread

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/edoardocomar/kafka KAFKA-3396

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1428.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1428


          commit 623f3ebd3ecca664a3b201fa86aa58964427f972
          Author: Edoardo Comar <ecomar@uk.ibm.com>
          Date: 2016-05-25T15:22:06Z

          KAFKA-3396 : Unauthorized topics are returned to the user

          Modified KafkaApis to return Errors.UNKNOWN_TOPIC_OR_PARTITION if
          principal has no Describe access to topic

          Unit tests expanded

          Some paths cause the client to block due to bug
          https://issues.apache.org/jira/browse/KAFKA-3727?filter=-2
          tests work around this by executing in separate thread


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user edoardocomar opened a pull request: https://github.com/apache/kafka/pull/1428 KAFKA-3396 : Unauthorized topics are returned to the user Modified KafkaApis to return Errors.UNKNOWN_TOPIC_OR_PARTITION if principal has no Describe access to topic Unit tests expanded Some paths cause the client to block due to bug https://issues.apache.org/jira/browse/KAFKA-3727?filter=-2 tests work around this by executing in separate thread You can merge this pull request into a Git repository by running: $ git pull https://github.com/edoardocomar/kafka KAFKA-3396 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1428 commit 623f3ebd3ecca664a3b201fa86aa58964427f972 Author: Edoardo Comar <ecomar@uk.ibm.com> Date: 2016-05-25T15:22:06Z KAFKA-3396 : Unauthorized topics are returned to the user Modified KafkaApis to return Errors.UNKNOWN_TOPIC_OR_PARTITION if principal has no Describe access to topic Unit tests expanded Some paths cause the client to block due to bug https://issues.apache.org/jira/browse/KAFKA-3727?filter=-2 tests work around this by executing in separate thread
          Show
          ecomar Edoardo Comar added a comment - I'm held back by https://issues.apache.org/jira/browse/KAFKA-3727 and https://issues.apache.org/jira/browse/KAFKA-3728
          Hide
          ecomar Edoardo Comar added a comment -

          Hi Grant Henke we're still working on it,
          we're working to get the unit tests pass (and added a few others) .

          We had some surprises running `SaslSslEndToEndAuthorizationTest`
          if we change the consumers from
          ```consumers.head.assign(List(tp).asJava)```
          to
          ```consumers.head.subscribe(List(topic).asJava) ```
          the code paths are different and even the original tests may not pass unless we change
          ``` this.serverConfig.setProperty(KafkaConfig.MinInSyncReplicasProp, "1")```
          to be a `"1"`
          which is the case also for the original code. Not sure if we're missing something.

          Show
          ecomar Edoardo Comar added a comment - Hi Grant Henke we're still working on it, we're working to get the unit tests pass (and added a few others) . We had some surprises running `SaslSslEndToEndAuthorizationTest` if we change the consumers from ```consumers.head.assign(List(tp).asJava)``` to ```consumers.head.subscribe(List(topic).asJava) ``` the code paths are different and even the original tests may not pass unless we change ``` this.serverConfig.setProperty(KafkaConfig.MinInSyncReplicasProp, "1")``` to be a `"1"` which is the case also for the original code. Not sure if we're missing something.
          Hide
          ecomar Edoardo Comar added a comment -

          Hi Grant Henke I thought that if a user has DESCRIBE permission on topic yet to be created,
          autocreate is on, but the user has no CREATE permission on cluster,
          there is no reason to hide the name of the topic or hide the reason of the failure.

          The fact that, in SimpleAclAuthorizer, the DESCRIBE permission only accepts the name of the topic or a wildcard for all topic,
          makes this scenario not very likely, IMHO, but still a possibility.
          Might be more common with an Authorizer implementation that allows topic names with wildcard
          (e.g. I could have DESCRIBE on all topics starting named like "edo-*")

          have to work on unit tests before I can make a PR

          Show
          ecomar Edoardo Comar added a comment - Hi Grant Henke I thought that if a user has DESCRIBE permission on topic yet to be created, autocreate is on, but the user has no CREATE permission on cluster, there is no reason to hide the name of the topic or hide the reason of the failure. The fact that, in SimpleAclAuthorizer, the DESCRIBE permission only accepts the name of the topic or a wildcard for all topic, makes this scenario not very likely, IMHO, but still a possibility. Might be more common with an Authorizer implementation that allows topic names with wildcard (e.g. I could have DESCRIBE on all topics starting named like "edo-*") have to work on unit tests before I can make a PR
          Hide
          granthenke Grant Henke added a comment -

          Edoardo Comar Thanks for working on a patch! Feel free to assign to yourself to this jira and send a PR. I am sure the exact rules may require some discussion.

          Can you help my understand why don't we want to return UNKNOWN_TOPIC_OR_PARTITION when auto-create topic is on, but the user has no CREATE permission on Cluster?

          FYI: We are slowly woking on moving auto-creation to be client side (KAFKA-2410), but that requires some of the KIP-4 work first. I am hoping to get that work done shortly after the 0.10 release.

          Show
          granthenke Grant Henke added a comment - Edoardo Comar Thanks for working on a patch! Feel free to assign to yourself to this jira and send a PR. I am sure the exact rules may require some discussion. Can you help my understand why don't we want to return UNKNOWN_TOPIC_OR_PARTITION when auto-create topic is on, but the user has no CREATE permission on Cluster? FYI: We are slowly woking on moving auto-creation to be client side ( KAFKA-2410 ), but that requires some of the KIP-4 work first. I am hoping to get that work done shortly after the 0.10 release.
          Hide
          ecomar Edoardo Comar added a comment -

          Hi
          we have worked on a simple patch that would return Errors.UNKNOWN_TOPIC_OR_PARTITION on a TopicMetadataRequest if the session's user has no DESCRIBE access to the Topic.

          we would only return TOPIC_AUTHORIZATION error if

          • the user has DESCRIBE
            AND
          • autocreate topic is on, but the user has no CREATE permission on Cluster.

          This should make an attacker, without permissions , unable to distinguish between a topic that exists and one that doesn't exist.

          Note that with Authenticator and ACL, the setting autocreate=true on the broker, would require autocreation of the ACL on the new topic,
          else a user is still unable to create and use a new topic.

          Edo and Mickael Maison

          Show
          ecomar Edoardo Comar added a comment - Hi we have worked on a simple patch that would return Errors.UNKNOWN_TOPIC_OR_PARTITION on a TopicMetadataRequest if the session's user has no DESCRIBE access to the Topic. we would only return TOPIC_AUTHORIZATION error if the user has DESCRIBE AND autocreate topic is on, but the user has no CREATE permission on Cluster. This should make an attacker, without permissions , unable to distinguish between a topic that exists and one that doesn't exist. Note that with Authenticator and ACL, the setting autocreate=true on the broker, would require autocreation of the ACL on the new topic, else a user is still unable to create and use a new topic. Edo and Mickael Maison

            People

            • Assignee:
              ecomar Edoardo Comar
              Reporter:
              granthenke Grant Henke
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development