[KAFKA-16966] Allow offset commit fetch to reuse previous request if partitions are a subset

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.8.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor
• Assignee: TaiJuWu
• Reporter: Kirk True
• Created: 14/Jun/24 18:34
• Updated: 07/Oct/24 04:02

Description

In initWithCommittedOffsetsIfNeeded, the behavior of the existing LegacyKafkaConsumer is to allow reuse only if the partitions for the current request equal those of the previous request exactly (source). That logic is the basis for the behavior used in the AsyncKafkaConsumer.

The proposed change is to allow for request reuse if the partitions for the current request are a subset of those of the previous request. This introduces a subtle difference in behavior between the two Consumer implementations, so we need to decided if we want to change both implementations or just AsyncKafkaConsumer. Also, the specific case that the request reuse logic solves is when the user has passed in a very low timeout value in a tight poll() loop, which suggests the partitions wouldn't be changing between those loops.