[KAFKA-17294] Handle retriable errors when fetching offsets in new consumer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.0, 3.8.0
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: kip-848-client-support
• Assignee: TengYao Chi
• Reporter: Lianet Magrans
• Created: 07/Aug/24 18:51
• Updated: 03/Sep/24 13:23
• Resolved: 03/Sep/24 11:05

Description

The new consumer CommitRequestManager fails with a fatal KafkaException whenever it receives an unexpected error in the OffsetFetch response, even if the error is retriable. See https://github.com/apache/kafka/blob/837684a1b9b3bad244613211e90b67cf9170fb44/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L1009-L1011

This was initially implemented like this to maintain the behaviour of the Classic consumer, where the ConsumerCoordinator would do the same when handling the OffsetFetchResponse. That behaviour is being updated for the legacy coordinator as part of KAFKA-17279, to retry on all retriable errors.

We should review and update in the commitRequestManager to align with this, and retry on all retriable errors, which seems sensible when fetching offsets. 

Issue Links

• relates to: KAFKA-17279 Handle retriable errors from offset fetches in ConsumerCoordinator
• links to: GitHub Pull Request #16833