Critical Description
Using the new consumer group protocol with kafka-console-consumer.sh, I find that if I leave the consumer with no records to process for 5 minutes (max.poll.interval.ms = 300000ms), the tool logs the following warning message and leaves the group.
"consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records."
With the older consumer, this did not occur.
The reason is that the consumer keeps a poll timer which is used to ensure liveness of the application thread. The poll timer automatically updates while the `Consumer.poll(Duration)` method is blocked, while the newer consumer only updates the poll timer when a new call to `Consumer.poll(Duration)` is issued. This means that the kafka-console-consumer.sh tools, which uses a very long timeout by default, works differently with the new consumer.
Attachments
Issue Links
- links to