Kafka
  1. Kafka
  2. KAFKA-1119

Kafka 0.8.1 overwrites previous per topic config changes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.1
    • Fix Version/s: None
    • Component/s: config
    • Labels:
      None

      Description

      kafka-topics --alter --config overwrites the previous per topic configs. There is no way to override more than one per topic config for the same topic

      1. KAFKA-1119_2013-11-07_17:07:18.patch
        9 kB
        Neha Narkhede
      2. KAFKA-1119_2013-11-07_10:17:14.patch
        8 kB
        Neha Narkhede
      3. KAFKA-1119_2013-11-07_09:50:20.patch
        7 kB
        Neha Narkhede
      4. KAFKA-1119_2013-11-06_10:13:13.patch
        5 kB
        Neha Narkhede
      5. KAFKA-1119.patch
        6 kB
        Neha Narkhede

        Activity

        Hide
        Neha Narkhede added a comment -

        Turns out the right way to use the existing tool to specify more than one override is -

        ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic topicconfigtest --alter --config retention.ms=180000 --config segment.ms=180000

        Show
        Neha Narkhede added a comment - Turns out the right way to use the existing tool to specify more than one override is - ./bin/kafka-topics.sh --zookeeper localhost:2181 --topic topicconfigtest --alter --config retention.ms=180000 --config segment.ms=180000
        Hide
        Jay Kreps added a comment -

        Hmm, but it looks like the current behavior is to just blindly set whatever properties are given on the command line as the FULL set of properties. So if I say
        ./bin/kafka-topics.sh --alter --config retention.ms=180000
        I might think I am just changing the retention.ms but in reality I am unsetting any previous per-topic configs (whatever they may be) and setting retention.ms.

        Currently the code looks like this:
        val configs = parseTopicConfigs(opts)
        AdminUtils.changeTopicConfig(zkClient, topic, configs)

        I think it might be more intuitive if we interpreted the args as a diff to apply to the current settings. So the code would be soemthing like:
        val configs = AdminUtils.fetchTopicConfig(zkClient, topic)
        configs.addAll(parseTopicConfigs(opts))
        AdminUtils.changeTopicConfig(zkClient, topic, configs)

        This does have a downside, though. How do you remove an override? Let's say there is a default of X, setting X actually sets a topic override of X which is different from having no setting for the config (because if you change the default that topic would be unaffected). This is not hugely important but some options for handling this would be
        1. No value means delete the config: --config x=
        2. Add a new option for unconfiguring: --config x=12 and --unconfig x

        Show
        Jay Kreps added a comment - Hmm, but it looks like the current behavior is to just blindly set whatever properties are given on the command line as the FULL set of properties. So if I say ./bin/kafka-topics.sh --alter --config retention.ms=180000 I might think I am just changing the retention.ms but in reality I am unsetting any previous per-topic configs (whatever they may be) and setting retention.ms. Currently the code looks like this: val configs = parseTopicConfigs(opts) AdminUtils.changeTopicConfig(zkClient, topic, configs) I think it might be more intuitive if we interpreted the args as a diff to apply to the current settings. So the code would be soemthing like: val configs = AdminUtils.fetchTopicConfig(zkClient, topic) configs.addAll(parseTopicConfigs(opts)) AdminUtils.changeTopicConfig(zkClient, topic, configs) This does have a downside, though. How do you remove an override? Let's say there is a default of X, setting X actually sets a topic override of X which is different from having no setting for the config (because if you change the default that topic would be unaffected). This is not hugely important but some options for handling this would be 1. No value means delete the config: --config x= 2. Add a new option for unconfiguring: --config x=12 and --unconfig x
        Hide
        Neha Narkhede added a comment -

        Yes, removing overrides is an issue today. I like option #1 where no value means delete the per-topic override.

        Show
        Neha Narkhede added a comment - Yes, removing overrides is an issue today. I like option #1 where no value means delete the per-topic override.
        Hide
        Neha Narkhede added a comment -

        Created reviewboard https://reviews.apache.org/r/15274/
        against branch trunk

        Show
        Neha Narkhede added a comment - Created reviewboard https://reviews.apache.org/r/15274/ against branch trunk
        Hide
        Neha Narkhede added a comment -

        Updated reviewboard https://reviews.apache.org/r/15274/
        against branch trunk

        Show
        Neha Narkhede added a comment - Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk
        Hide
        Neha Narkhede added a comment -

        Updated reviewboard https://reviews.apache.org/r/15274/
        against branch trunk

        Show
        Neha Narkhede added a comment - Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk
        Hide
        Neha Narkhede added a comment -

        Updated reviewboard https://reviews.apache.org/r/15274/
        against branch trunk

        Show
        Neha Narkhede added a comment - Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk
        Hide
        Neha Narkhede added a comment -

        Updated reviewboard https://reviews.apache.org/r/15274/
        against branch trunk

        Show
        Neha Narkhede added a comment - Updated reviewboard https://reviews.apache.org/r/15274/ against branch trunk
        Hide
        Neha Narkhede added a comment -

        Thanks for the reviews! Pushed to trunk.

        Show
        Neha Narkhede added a comment - Thanks for the reviews! Pushed to trunk.

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development