Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.10.0.1
-
None
-
None
Description
In KafkaConsumer:poll, the code always calls "pollNoWakeup", which turns out to be expensive. When max.poll.records=1, for example, that call adds about 50% performance overhead. The code should avoid avoid that function unnecessarily when there are no outstanding prefetches.
Old JIRA description (discarded because turned out not to be the case):
---------------------------------------------------------------------------------------------------
Now kafka consumer has added max.poll.records to limit the count of messages return by poll().
According to KIP-41, to implement max.poll.records, the prefetch request should only be sent when the total number of retained records is less than max.poll.records.
But in the code of 0.10.0.1 , the consumer will send a prefetch request if it retained any records and never check if total number of retained records is less than max.poll.records..
If max.poll.records is set to a count much less than the count of message fetched , the poll() loop will send a lot of requests than expected and will have more and more records fetched and stored in memory before they can be consumed.
So before sending a prefetch request , the consumer must check if total number of retained records is less than max.poll.records.
Attachments
Issue Links
- links to
