[KAFKA-10254] 100% cpu usage by kafkaConsumer poll , when broker can't be connect

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.5.1
• Component/s: clients
• Assignee: Unassigned
• Reporter: xiaotong.wang
• Created: 09/Jul/20 11:29
• Updated: 17/Jul/20 16:47
• Resolved: 17/Jul/20 16:27

Description

steps

1、start kafka broker 

2、start kafka consumer and subscribe some topic with some kafkaConsumer instance and  call  kafkaConsumer.poll(Duration.ofMillis(pollTimeout))   and set auto.commit.enabled=false

3、iptables to disable kafka broker  ip  in client vm or shutdown kafka brokers

4、cpu go to 100%

 

why?

 

 

left Vserison :2.3.1

right Version:2.5.0

 

for 2.3.1 kafkaConsumer when kafka  brokers go  down,updateAssignmentMetadataIfNeeded will block x ms and return empty records ,

 

for 2.5.0

private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollForFetches(Timer timer) {
long pollTimeout = coordinator == null ? timer.remainingMs() :
Math.min(coordinator.timeToNextPoll(timer.currentTimeMs()), timer.remainingMs());

i check the source of kafka client ,poll timeout will be change to 0 ms ,when heartbeat timeout ，so  it will call poll without any block ,this will cause cpu go to 100%

 

 

 

Issue Links

• duplicates: KAFKA-10134 High CPU issue during rebalance in Kafka consumer after upgrading to 2.5