[KAFKA-10518] Consumer fetches could be inefficient when lags are unbalanced

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consumer
• Assignee: Unassigned
• Reporter: Dhruvil Shah
• Created: 23/Sep/20 22:15
• Updated: 16/Mar/21 06:17

Description

Consumer fetches are inefficient when lags are imbalanced across partitions, due to head of the line blocking and the behavior of blocking for `max.wait.ms` until data is available.

When the consumer receives a fetch response, it prepares the next fetch request and sends it out. The caveat is that the subsequent fetch request would explicitly exclude partitions for which the consumer received data in the previous round. This is to allow the consumer application to drain the data for those partitions, until the consumer fetches the other partitions it is subscribed to.

This behavior does not play out too well if the consumer is consuming when the lag is unbalanced, because it would receive data for the partitions it is lagging on, and then it would send a fetch request for partitions that do not have any data (or have little data). The latter will end up blocking for fetch.max.wait.ms on the broker before an empty response is sent back. This slows down the consumer’s overall consumption throughput since fetch.max.wait.ms is 500ms by default.

Issue Links

• is related to: KAFKA-10517 Inefficient consumer processing with fetch sessions