[KAFKA-15441] Broker sessions can time out during ZK migration

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.6.0
• Fix Version/s: 3.6.0
• Assignee: David Arthur
• Reporter: David Arthur
• Created: 06/Sep/23 14:16
• Updated: 08/Sep/23 21:58
• Resolved: 08/Sep/23 21:58

Description

When a ZK to KRaft migration takes more than a few seconds to complete, the sessions between the ZK brokers and the KRaft controller will expire. This appears to be due to the heartbeat events being blocked in the purgatory on the controller.

The side effect of this expiration is that after the metadata is migrated, the KRaft controller will immediately fence all of the brokers and remove them from ISRs. This leads to a mass leadership change that can cause large latency spikes on the brokers.

Issue Links

• links to: GitHub Pull Request #14351