[KAFKA-4007] Improve fetch pipelining for low values of max.poll.records

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer
• Assignee: Mickael Maison
• Reporter: Jason Gustafson
• Created: 29/Jul/16 22:16
• Updated: 30/Nov/16 09:43

Description

Currently the consumer will only send a prefetch for a partition after all the records from the previous fetch have been consumed. This can lead to suboptimal pipelining when max.poll.records is set very low since the processing latency for a small set of records may be small compared to the latency of a fetch. An improvement suggested by junrao is to send the fetch anyway even if we have unprocessed data buffered, but delay reading it from the socket until that data has been consumed. Potentially the consumer can delay reading any pending fetch until it is ready to be returned to the user, which may help control memory better.