Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4007

Improve fetch pipelining for low values of max.poll.records

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: consumer
    • Labels:
      None

      Description

      Currently the consumer will only send a prefetch for a partition after all the records from the previous fetch have been consumed. This can lead to suboptimal pipelining when max.poll.records is set very low since the processing latency for a small set of records may be small compared to the latency of a fetch. An improvement suggested by Jun Rao is to send the fetch anyway even if we have unprocessed data buffered, but delay reading it from the socket until that data has been consumed. Potentially the consumer can delay reading any pending fetch until it is ready to be returned to the user, which may help control memory better.

        Attachments

          Activity

            People

            • Assignee:
              mimaison Mickael Maison
              Reporter:
              hachikuji Jason Gustafson
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: