Minor Description
Upon restarting kafka, dynamically configured log cleaner configs aren't picked up and applied.
Here are some logs from a local kafka when I up the threads to 2 using the kafka-config tool - Noting the last 2 lines where it starts up 2 log cleaner threads.
[2021-11-23 21:09:50,044] INFO [Admin Manager on Broker 1001]: Updating brokers with new configuration : log.cleaner.threads -> 2 (kafka.server.ZkAdminManager) [2021-11-23 21:09:50,092] INFO Processing override for entityPath: brokers/<default> with config: HashMap(log.cleaner.threads -> 2) (kafka.server.DynamicConfigManager) log.cleaner.threads = 2 [2021-11-23 21:09:50,113] INFO Shutting down the log cleaner. (kafka.log.LogCleaner) [2021-11-23 21:09:50,114] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner) [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner) [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner) [2021-11-23 21:09:50,119] INFO Starting the log cleaner (kafka.log.LogCleaner) [2021-11-23 21:09:50,178] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner) [2021-11-23 21:09:50,181] INFO [kafka-log-cleaner-thread-1]: Starting (kafka.log.LogCleaner)
And now after a restart, at no point does it ever start 2 threads, even though it clearly knows about the configs
[2021-11-23 21:10:46,659] INFO Starting the log cleaner (kafka.log.LogCleaner) [2021-11-23 21:10:46,723] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner) [2021-11-23 21:10:48,124] INFO Processing override for entityPath: brokers/<default> with config: HashMap(log.cleaner.threads -> 2) (kafka.server.DynamicConfigManager) log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 150000000 log.cleaner.delete.retention.ms = 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 2
When investigating from the kafka config tool all looks well.
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers --entity-default --describe --all | grep log.cleaner.threads log.cleaner.threads=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}
But if you try change the config you soon find out that all is not well(note here it mentions the current value is 1 in the validation message)
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=3 Error while executing config command with args '--bootstrap-server profile_kafka:9093 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=3' java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name=''): Invalid value org.apache.kafka.common.config.ConfigException: Log cleaner threads cannot be increased to more than double the current value 1 for configuration Invalid dynamic configuration