[KAFKA-15605] Topics marked for deletion in ZK are incorrectly migrated to KRaft

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.7.0, 3.6.1
• Component/s: controller, kraft
• Assignee: David Arthur
• Reporter: David Arthur
• Created: 13/Oct/23 19:24
• Updated: 14/Nov/23 18:19
• Resolved: 14/Nov/23 18:19

Description

When migrating topics from ZooKeeper, the KRaft controller reads all the topic and partition metadata from ZK directly. This includes topics which have been marked for deletion by the ZK controller. After being migrated to KRaft, the pending topic deletions are never completed, so it is as if the delete topic request never happened.

Since the client request to delete these topics has already been returned as successful, it would be confusing to the client that the topic still existed. An operator or application would need to issue another topic deletion to remove these topics once the controller had moved to KRaft. If they tried to create a new topic with the same name, they would receive a TOPIC_ALREADY_EXISTS error.

The migration logic should carry over pending topic deletions and resolve them either as part of the migration or shortly after.

Note to operators:
To determine if a migration was affected by this, an operator can check the contents of /admin/delete_topics after the KRaft controller has migrated the metadata. If any topics are listed under this ZNode, they were not deleted and will still be present in KRaft. At this point the operator can make a determination if the topics should be re-deleted (using "kafka-topics.sh --delete") or left in place. In either case, the topics should be removed from /admin/delete_topics to prevent unexpected topic deletion in the event of a fallback to ZK.

Issue Links

• causes: KAFKA-15793 Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions
• links to: GitHub Pull Request #14545