[KAFKA-12980] Allow consumers to return from poll when position advances due to aborted transactions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: consumer
• Assignee: Chris Egerton
• Reporter: Chris Egerton
• Created: 22/Jun/21 17:38
• Updated: 29/Mar/23 04:46
• Resolved: 09/Dec/21 04:37

Description

When Consumer::poll is invoked on a topic with an open transaction, and then that transaction is aborted, poll does not return unless there are other records available in that topic after the aborted transaction.

Instead, poll could return in this case, even when no records are available.

This facilitates reads to the end of a topic where the end offsets of a topic are listed and then a consumer for that topic is polled until its position is at or beyond each of those offsets (for example, Connect does this when reading to the end of any of its internal topics).

We could update the existing language in the docs for Consumer::poll from

This method returns immediately if there are records available.

to

This method returns immediately if there are records available or if the position advances past control records.

 

A workaround for existing users who would like to see this is to use short poll intervals and manually check the consumer's position in between each poll, but this is fairly tedious and may lead to excess CPU and network utilization depending on the latency requirements for knowing when the end of the topic has been reached.

Issue Links

• causes: KAFKA-14865 Consumer::poll returns early on with an empty result for highly compacted KTables
• fixes: KAFKA-14713 Kafka Streams global table startup takes too long
• fixes: KAFKA-14442 GlobalKTable restoration waits requestTimeout during application restart
• relates to: KAFKA-12951 Infinite loop while restoring a GlobalKTable
• relates to: KAFKA-6607 Kafka Streams lag not zero when input topic transactional
• links to: GitHub Pull Request #11046