[KAFKA-10726] How to detect heartbeat failure between broker/zookeeper leader

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Information Provided
• Affects Version/s: 2.1.1
• Component/s: controller, logging
• Assignee: Unassigned
• Reporter: Keiichiro Wakasa
• Created: 16/Nov/20 10:11
• Updated: 09/Oct/24 09:13
• Resolved: 09/Oct/24 09:13

Description

Hello experts,

I'm not sure this is proper place to ask but I'd appreciate if you could help us with the following question...

 

We've continuously suffered from broker exclusion caused by heartbeat timeout between broker and zookeeper leader.

This issue can be easily detected by checking ephemeral nodes via zkcli.sh but we'd like to detect this with logs like server.log/controller.log since we have an existing system to forward these logs to our system. 

Looking at server.log/controller.log, we couldn't find any logs that indicates the heartbeat timeout. Is there any other logs to check for heartbeat health?