[KAFKA-17674] New consumer reset positions for newly added partitions before retrieving committed offsets

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848-client-support
• Assignee: Lianet Magrans
• Reporter: Lianet Magrans
• Reviewer: Chia-Ping Tsai
• Created: 5 days ago 15:28
• Updated: 3 days ago 11:12
• Resolved: 3 days ago 11:12

Description

The update fetch positions flow in new consumer attempts to :

1. retrieve committed offsets
2. reset positions for partitions that may still require one.

In the case that a partition with committed offsets is assigned in-between 1 and 2, the new consumer mistakenly included the newly added partition in the LIST_OFFSETS request issued as part of 2 (so it updates the new partition position using the partition offset and not the committed offsets)

We should ensure that we preserve the initial set of partitions we're initializing, so that the reset excludes partitions that may require a positions but that are not in the initial set. Those partitions should get a position on the next poll iteration (going through steps 1 and 2)

Issue Links

• links to: PR