Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4348

On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.9.0.0, 0.9.0.1, 0.10.0.1
    • Fix Version/s: None
    • Component/s: consumer
    • Labels:
    • Environment:
      Mac OS X EI Capitan, Java 1.8.0_111

      Description

      Steps to reproduce:
      1. start the zookeeper and kafka server using the default properties from the distribution:
      $ bin/zookeeper-server-start.sh config/zookeeper.properties
      $ bin/kafka-server-start.sh config/server.properties

      2. create a Kafka consumer using the Java API KafkaConsumer.poll(long timeout). It polls the records from the server every second (timeout set to 1000) and prints the number of records polled. The code can be found here: https://gist.github.com/yiquanzhou/a94569a2c4ec8992444c83f3c393f596

      3. use bin/kafka-verifiable-producer.sh to generate some messages:
      $ bin/kafka-verifiable-producer.sh --topic connect-test --max-messages 200000 --broker-list localhost:9092
      wait until all 200k messages are generated and sent to the server.

      4. Run the consumer Java code. In the output console of the consumer, we can see that the consumer starts to poll some records, then it polls 0 records for several seconds before polling some more. like this:

      polled 27160 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 26886 records
      polled 26886 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 0 records
      polled 26701 records
      polled 26214 records

      The bug slows down the consumption of messages a lot. And in our use case, the consumer wrongly assumes that all messages are read from the topic.

      It is only reproducible on Mac OS X but neither on Linux nor Windows.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                huxi_2b huxihx
                Reporter:
                yqzhou Yiquan Zhou
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: