[KAFKA-16938] non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.9.0
• Assignee: TengYao Chi
• Reporter: Chia-Ping Tsai
• Created: 12/Jun/24 08:24
• Updated: 13/Jun/24 01:48
• Resolved: 13/Jun/24 01:48

Description

DynamicBrokerConfig has circular reference with DynamicConfig. The following initialization order will cause incorrect non-dynamic props [0]

1. DynamicConfig is initializing -> brokerConfigs is created [1]
2. DynamicConfig is initializing -> call DynamicBrokerConfig.addDynamicConfigs(brokerConfigs) [2]
3. DynamicBrokerConfig is initializing -> nonDynamicProps: Set[String] = KafkaConfig.configNames.toSet – DynamicConfig.Broker.names.asScala [3]
4. DynamicConfig.Broker.names reference `brokerConfigs`, and `brokerConfigs` does not have all dynamic props (step2), so nonDynamicProps get created with incorrect contents.

We should break the circular by addressing following tasks:

1. move `DynamicBrokerConfig.addDynamicConfigs` to `DynamicConfig.Broker`
2. move `DynamicBrokerConfig#nonDynamicProps` to `DynamicConfig.Broker`

object DynamicConfig {

  object Broker {
    private val brokerConfigs = {
      val configs = QuotaConfigs.brokerQuotaConfigs()
      KafkaConfig.configKeys.filter {
        case (configName, _) => AllDynamicConfigs.contains(configName)
      }.foreach {
        case (_, config) =>
          configs.define(config.name, config.`type`, config.defaultValue, config.validator,
            config.importance, config.documentation, config.group, config.orderInGroup, config.width,
            config.displayName, config.dependents, config.recommender)
      }
      configs
    }

    val nonDynamicProps: Set[String] = KafkaConfig.configNames.toSet -- brokerConfigs.names.asScala

[0] https://github.com/apache/kafka/blob/638844f833b165d6f9ca52c173858d26b7254fac/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L120
[1] https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicConfig.scala#L35
[2] https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicConfig.scala#L36
[3] https://github.com/apache/kafka/blob/638844f833b165d6f9ca52c173858d26b7254fac/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L120

Issue Links

• links to: GitHub Pull Request #16302