Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3135

Unexpected delay before fetch response transmission

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
    • None
    • None
    • None

    Description

      From the user list, Krzysztof Ciesielski reports the following:

      Scenario description:
      First, a producer writes 500000 elements into a topic
      Then, a consumer starts to read, polling in a loop.
      When "max.partition.fetch.bytes" is set to a relatively small value, each
      "consumer.poll()" returns a batch of messages.
      If this value is left as default, the output tends to look like this:

      Poll returned 13793 elements
      Poll returned 13793 elements
      Poll returned 13793 elements
      Poll returned 13793 elements
      Poll returned 0 elements
      Poll returned 0 elements
      Poll returned 0 elements
      Poll returned 0 elements
      Poll returned 13793 elements
      Poll returned 13793 elements

      As we can see, there are weird "gaps" when poll returns 0 elements for some
      time. What is the reason for that? Maybe there are some good practices
      about setting "max.partition.fetch.bytes" which I don't follow?

      The gist to reproduce this problem is here: https://gist.github.com/kciesielski/054bb4359a318aa17561.

      After some initial investigation, the delay appears to be in the server's networking layer. Basically I see a delay of 5 seconds from the time that Selector.send() is invoked in SocketServer.Processor with the fetch response to the time that the send is completed. Using netstat in the middle of the delay shows the following output:

      tcp4       0      0  10.191.0.30.55455      10.191.0.30.9092       ESTABLISHED
      tcp4       0 102400  10.191.0.30.9092       10.191.0.30.55454      ESTABLISHED
      

      From this, it looks like the data reaches the send buffer, but needs to be flushed.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hachikuji Jason Gustafson
            hachikuji Jason Gustafson

            Dates

              Created:
              Updated:

              Slack

                Issue deployment