Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10254

100% cpu usage by kafkaConsumer poll , when broker can't be connect

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.5.1
    • Component/s: clients
    • Labels:
      None

      Description

      steps

      1、start kafka broker 

      2、start kafka consumer and subscribe some topic with some kafkaConsumer instance and  call  kafkaConsumer.poll(Duration.ofMillis(pollTimeout))   and set auto.commit.enabled=false

      3、iptables to disable kafka broker  ip  in client vm or shutdown kafka brokers

      4、cpu go to 100%

       

      why?

       

       

      left Vserison :2.3.1

      right Version:2.5.0

       

      for 2.3.1 kafkaConsumer when kafka  brokers go  down,updateAssignmentMetadataIfNeeded will block x ms and return empty records ,

       

      for 2.5.0

      private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollForFetches(Timer timer) {
      long pollTimeout = coordinator == null ? timer.remainingMs() :
      Math.min(coordinator.timeToNextPoll(timer.currentTimeMs()), timer.remainingMs());

      i check the source of kafka client ,poll timeout will be change to 0 ms ,when heartbeat timeout ,so  it will call poll without any block ,this will cause cpu go to 100%

       

       

       

        Attachments

        1. image-2020-07-09-19-24-20-604.png
          52 kB
          xiaotong.wang

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xiaotong.wang xiaotong.wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: