[KAFKA-17608] KRaft controller crashes when leader is removed from quorum

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.9.0
• Fix Version/s: 3.9.0
• Component/s: controller
• Assignee: Alyssa Huang
• Reporter: José Armando García Sancio
• Created: 25/Sep/24 17:39
• Updated: 26/Sep/24 22:19
• Resolved: 26/Sep/24 22:19

Description

After removing the active controller with:

bin/kafka-metadata-quorum.sh --command-config config/kraft/controller2.properties --bootstrap-controller localhost:9093 remove-controller --controller-id 1 --controller-directory-id XvLXb0JPTSKRfsy9t47l-Q
Removed  KRaft controller 1 with directory id XvLXb0JPTSKRfsy9t47l-Q

The active controller fails with:

[2024-09-25 17:35:35,287] ERROR Encountered fatal fault: exception while renouncing leadership (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
java.lang.RuntimeException: Unable to reset to last stable offset 11. No in-memory snapshot found for this offset.
        at org.apache.kafka.controller.OffsetControlManager.deactivate(OffsetControlManager.java:268)
        at org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1281)
        at org.apache.kafka.controller.QuorumController$QuorumMetaLogListener.lambda$handleLeaderChange$2(QuorumController.java:1177)
        at org.apache.kafka.controller.QuorumController$QuorumMetaLogListener.lambda$appendRaftEvent$3(QuorumController.java:1201)
        at org.apache.kafka.controller.QuorumController$ControllerEvent.run(QuorumController.java:585)
        at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:132)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215)
        at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186)
        at java.base/java.lang.Thread.run(Thread.java:840)
 

Issue Links

• links to: GitHub Pull Request #17146