Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13473

Log cleaner Dynamic configs aren't applied after a restart



    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 2.8.1
    • None
    • config, core
    • None


      Upon restarting kafka, dynamically configured log cleaner configs aren't picked up and applied.

      Here are some logs from a local kafka when I up the threads to 2 using the kafka-config tool - Noting the last 2 lines where it starts up 2 log cleaner threads.


      [2021-11-23 21:09:50,044] INFO [Admin Manager on Broker 1001]: Updating brokers with new configuration : log.cleaner.threads -> 2 (kafka.server.ZkAdminManager)
      [2021-11-23 21:09:50,092] INFO Processing override for entityPath: brokers/<default> with config: HashMap(log.cleaner.threads -> 2) (kafka.server.DynamicConfigManager) log.cleaner.threads = 2
      [2021-11-23 21:09:50,113] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,114] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,119] INFO Starting the log cleaner (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,178] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
      [2021-11-23 21:09:50,181] INFO [kafka-log-cleaner-thread-1]: Starting (kafka.log.LogCleaner)

      And now after a restart, at no point does it ever start 2 threads, even though it clearly knows about the configs


      [2021-11-23 21:10:46,659] INFO Starting the log cleaner (kafka.log.LogCleaner)
      [2021-11-23 21:10:46,723] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
      [2021-11-23 21:10:48,124] INFO Processing override for entityPath: brokers/<default> with config: HashMap(log.cleaner.threads -> 2) (kafka.server.DynamicConfigManager) log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 150000000 log.cleaner.delete.retention.ms = 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 2


      When investigating from the kafka config tool all looks well.

      kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers --entity-default --describe --all | grep log.cleaner.threads log.cleaner.threads=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}


      But if you try change the config you soon find out that all is not well(note here it mentions the current value is 1 in the validation message)

      kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=3 Error while executing config command with args '--bootstrap-server profile_kafka:9093 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=3' java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name=''): Invalid value org.apache.kafka.common.config.ConfigException: Log cleaner threads cannot be increased to more than double the current value 1 for configuration Invalid dynamic configuration





            Unassigned Unassigned
            tim.patterson Tim Patterson
            1 Vote for this issue
            3 Start watching this issue