[KAFKA-16583] Update from 3.4.0 to 3.7.0 image write failed in Kraft mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0, 3.7.1
• Component/s: kraft
• Flags: Patch
• Assignee: HanXu
• Reporter: HanXu
• Created: 18/Apr/24 07:01
• Updated: 05/Jun/24 07:06
• Resolved: 04/Jun/24 14:40

Description

How to reproduce:
1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode;
2. Create a topic with 1 partition;
3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 partition to Broker B;
4. Upgrade Broker B to 3.7.0;

The Broker B will keep log the following error:

[2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler)
org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.4-IV0: the directory assignment state of one or more replicas
	at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94)
	at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391)
	at org.apache.kafka.image.TopicImage.write(TopicImage.java:71)
	at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84)
	at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155)
	at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295)
	at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
	at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
	at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
	at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
	at java.base/java.lang.Thread.run(Thread.java:840)

Bug:

• When reassigning partition, PartitionRegistration#merge will set the new replicas with UNASSIGNED directory;
• But in metadata version 3.4.0 PartitionRegistration#toRecord only allows MIGRATING directory；

          if (options.metadataVersion().isDirectoryAssignmentSupported()) {
              record.setDirectories(Uuid.toList(directories));
          } else {
              for (Uuid directory : directories) {
                  if (!DirectoryId.MIGRATING.equals(directory)) {
                      options.handleLoss("the directory assignment state of one or more replicas");
                      break;
                  }
              }
          }
  

Solution:

• PartitionRegistration#toRecord allows both MIGRATING and UNASSIGNED

Issue Links

• is duplicated by: KAFKA-16662 UnwritableMetadataException: Metadata has been lost
• links to: GitHub Pull Request #15751
• links to: GitHub Pull Request #16118