Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
-
None
-
None
-
None
Description
From the user list, Krzysztof Ciesielski reports the following:
Scenario description:
First, a producer writes 500000 elements into a topic
Then, a consumer starts to read, polling in a loop.
When "max.partition.fetch.bytes" is set to a relatively small value, each
"consumer.poll()" returns a batch of messages.
If this value is left as default, the output tends to look like this:Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 13793 elements
Poll returned 13793 elementsAs we can see, there are weird "gaps" when poll returns 0 elements for some
time. What is the reason for that? Maybe there are some good practices
about setting "max.partition.fetch.bytes" which I don't follow?
The gist to reproduce this problem is here: https://gist.github.com/kciesielski/054bb4359a318aa17561.
After some initial investigation, the delay appears to be in the server's networking layer. Basically I see a delay of 5 seconds from the time that Selector.send() is invoked in SocketServer.Processor with the fetch response to the time that the send is completed. Using netstat in the middle of the delay shows the following output:
tcp4 0 0 10.191.0.30.55455 10.191.0.30.9092 ESTABLISHED tcp4 0 102400 10.191.0.30.9092 10.191.0.30.55454 ESTABLISHED
From this, it looks like the data reaches the send buffer, but needs to be flushed.
Attachments
Attachments
Issue Links
- is duplicated by
-
KAFKA-4348 On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on Kafka server
- Resolved