Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3741

Allow setting of default topic configs via StreamsConfig

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.10.0.0
    • 1.0.0
    • streams

    Description

      Kafka Streams currently allows you to specify a replication factor for changelog and repartition topics that it creates. It should also allow you to specify any other TopicConfig. These should be used as defaults when creating Internal topics. The defaults should be overridden by any configs provided by the StateStoreSuppliers etc.

      Attachments

        Activity

          githubbot ASF GitHub Bot added a comment -

          GitHub user dguy opened a pull request:

          https://github.com/apache/kafka/pull/3255

          KAFKA-3741: add min.insync.replicas config to Streams

          Allow users to specify `min.insync.replicas` via StreamsConfig. Default to `null` so that the server settting will be used. If `replication.factor` is lower than `min.insync.replicas` then set `min.insync.replicas` to `replication.factor`

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/dguy/kafka kafka-3741

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/3255.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3255


          commit a5f490d0921f5e929eb795b88c7f9b65233c69c7
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2017-06-07T11:04:08Z

          add min.insync.replicas config


          githubbot ASF GitHub Bot added a comment - GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3255 KAFKA-3741 : add min.insync.replicas config to Streams Allow users to specify `min.insync.replicas` via StreamsConfig. Default to `null` so that the server settting will be used. If `replication.factor` is lower than `min.insync.replicas` then set `min.insync.replicas` to `replication.factor` You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka kafka-3741 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3255.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3255 commit a5f490d0921f5e929eb795b88c7f9b65233c69c7 Author: Damian Guy <damian.guy@gmail.com> Date: 2017-06-07T11:04:08Z add min.insync.replicas config
          githubbot ASF GitHub Bot added a comment -

          Github user dguy closed the pull request at:

          https://github.com/apache/kafka/pull/3255

          githubbot ASF GitHub Bot added a comment - Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/3255
          githubbot ASF GitHub Bot added a comment -

          GitHub user dguy opened a pull request:

          https://github.com/apache/kafka/pull/3459

          KAFKA-3741: allow users to specify default topic configs for internal topics

          Allow users to specify default topic configs for streams internal topics by supplying properties from `TopicConfig` with a prefix.
          Supplied defaults are used when creating the internal topics. They are overridden by the configs supplied along with the `InternalTopicConfig`

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/dguy/kafka kafka-3741

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/3459.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3459


          commit 415e3130ae77aceefc2d3209b95e32178ba39348
          Author: Damian Guy <damian.guy@gmail.com>
          Date: 2017-06-29T15:34:52Z

          allow users to specify default topic configs with a prefix via StreamsConfig


          githubbot ASF GitHub Bot added a comment - GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3459 KAFKA-3741 : allow users to specify default topic configs for internal topics Allow users to specify default topic configs for streams internal topics by supplying properties from `TopicConfig` with a prefix. Supplied defaults are used when creating the internal topics. They are overridden by the configs supplied along with the `InternalTopicConfig` You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka kafka-3741 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3459.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3459 commit 415e3130ae77aceefc2d3209b95e32178ba39348 Author: Damian Guy <damian.guy@gmail.com> Date: 2017-06-29T15:34:52Z allow users to specify default topic configs with a prefix via StreamsConfig
          damianguy Damian Guy added a comment -

          Issue resolved by pull request 3459
          https://github.com/apache/kafka/pull/3459

          damianguy Damian Guy added a comment - Issue resolved by pull request 3459 https://github.com/apache/kafka/pull/3459
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/3459

          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3459
          guozhang Guozhang Wang added a comment -

          damianguy Could you update the upgrade guide web doc section for this change as well?

          guozhang Guozhang Wang added a comment - damianguy Could you update the upgrade guide web doc section for this change as well?
          wushujames James Cheng added a comment -

          Some of these docs seem to have disappeared from top of trunk, or even 1.0.0. In particular, the stuff about how to use the TOPIC_PREFIX stuff (see https://github.com/dguy/kafka/blob/9cd0e0fbb5c972151ba0081e8f1c366db5bd12e8/docs/streams/developer-guide.html#L498-L507)

          Does anyone know where those docs have gone to?

          wushujames James Cheng added a comment - Some of these docs seem to have disappeared from top of trunk, or even 1.0.0. In particular, the stuff about how to use the TOPIC_PREFIX stuff (see https://github.com/dguy/kafka/blob/9cd0e0fbb5c972151ba0081e8f1c366db5bd12e8/docs/streams/developer-guide.html#L498-L507) Does anyone know where those docs have gone to?

          Not sure... joel@confluent.io Can you have a look into this?

          mjsax Matthias J. Sax added a comment - Not sure... joel@confluent.io Can you have a look into this?

          People

            damianguy Damian Guy
            theduderog Roger Hoover
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: