[KAFKA-15553] Review consumer positions update

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: clients, consumer
• Labels: consumer-threading-refactor position
• Assignee: Philip Nee
• Reporter: Philip Nee
• Created: 05/Oct/23 19:27
• Updated: 07/Oct/24 16:40

Description

From the existing comment: If there are any partitions which do not have a valid position and are not awaiting reset, then we need to fetch committed offsets.

In the async consumer: I wonder if it would make sense to refresh the position on the event loop continuously.

The logic to refresh offsets in the poll loop is quite fragile and works largely by side-effects of the code that it calls. For example, the behaviour of the "cached" value is really not that straightforward and simply reading the cached value is not sufficient to start consuming data in all cases.

This area needs a bit of a refactor.

Issue Links

• is related to: KAFKA-16156 System test failing for new consumer on endOffsets with negative timestamps