[KAFKA-16969] KRaft unable to upgrade to v3.7.1 and later when multiple log dir is set

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.8.0, 3.7.1
• Fix Version/s: 3.8.0, 3.7.1
• Assignee: Igor Soarez
• Reporter: Luke Chen
• Created: 15/Jun/24 08:11
• Updated: 18/Jun/24 20:42
• Resolved: 18/Jun/24 03:14

Description

After KAFKA-16606, we added validation metadata version for JBOD support. This validation works well in isolated KRaft mode (i.e. separate controller/broker node). But when in combined mode, this validation will let the node fail to startup. The log will be like this:

[2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the broker metadata publishers to be installed (kafka.server.BrokerServer)
[2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Finished waiting for the broker metadata publishers to be installed (kafka.server.BrokerServer)
[2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the controller to acknowledge that we are caught up (kafka.server.BrokerServer)
[2024-06-15 16:00:45,621] INFO [MetadataLoader id=1] InitializeNewPublishers: initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4 (org.apache.kafka.image.loader.MetadataLoader)
[2024-06-15 16:00:45,621] ERROR Encountered metadata publishing fault: Broker configuration does not support the cluster MetadataVersion (org.apache.kafka.server.fault.LoggingFaultHandler)
java.lang.IllegalArgumentException: requirement failed: Multiple log directories (aka JBOD) are not supported in the current MetadataVersion 3.6-IV2. Need 3.7-IV2 or higher
    at scala.Predef$.require(Predef.scala:337)
    at kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545)
    at kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62)
    at kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55)
    at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309)
    at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
    at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
    at java.base/java.lang.Thread.run(Thread.java:1623)
[2024-06-15 16:00:45,622] ERROR Encountered fatal fault: Unhandled error initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4 (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
org.apache.kafka.server.fault.FaultHandlerException: Broker configuration does not support the cluster MetadataVersion
    at scala.Predef$.require(Predef.scala:337)
    at kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545)
    at kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62)
    at kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55)
    at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309)
    at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
    at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
    at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.IllegalArgumentException: requirement failed: Multiple log directories (aka JBOD) are not supported in the current MetadataVersion 3.6-IV2. Need 3.7-IV2 or higher
    ... 10 more

 

This will block combined node setting multiple log dirs upgrade to v3.7.1 or later.

Issue Links

• fixes: KAFKA-16961 TestKRaftUpgrade system tests fail in v3.7.1 RC1
• is caused by: KAFKA-16961 TestKRaftUpgrade system tests fail in v3.7.1 RC1
• links to: GitHub Pull Request #16352
• links to: GitHub Pull Request #16366