[KAFKA-15825] KRaft controller writes empty state to ZK after migration

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.7.0, 3.6.1
• Component/s: controller, kraft
• Assignee: David Arthur
• Reporter: David Arthur
• Created: 14/Nov/23 18:30
• Updated: 14/Nov/23 19:40
• Resolved: 14/Nov/23 18:31

Description

Immediately following the ZK migration, there is a race condition where the KRaftMigrationDriver can use an empty MetadataImage when performing the full "SYNC_KRAFT_TO_ZK" reconciliation.

After the next controller failover, or when the controller loads a metadata snapshot, the correct state will be written to ZK.

The symptom of this bug is that we see the migration complete, and then all the metadata removed from ZK. For example,

[KRaftMigrationDriver id=9990] Completed migration of metadata from ZooKeeper to KRaft. 573 records were generated in 2204 ms across 51 batches. The record types were {TOPIC_RECORD=41, PARTITION_RECORD=410, CONFIG_RECORD=121, PRODUCER_IDS_RECORD=1}. The current metadata offset is now 503794 with an epoch of 21. Saw 6 brokers in the migrated metadata [0, 1, 2, 3, 4, 5].

immediately followed by:

[KRaftMigrationDriver id=9990] Made the following ZK writes when reconciling with KRaft state: {DeleteBrokerConfig=7, DeleteTopic=41, UpdateTopicConfig=41}

If affected by this, a quick workaround is to cause the controller to failover.