[KAFKA-17470] CommitRequestManager should record failed request only once even if multiple errors in response

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: clients, consumer
• Labels: consumer-threading-refactor kip-848-client-support
• Assignee: TengYao Chi
• Reporter: Lianet Magrans
• Reviewer: Lianet Magrans
• Created: 03/Sep/24 13:22
• Updated: 09/Sep/24 19:53
• Resolved: 09/Sep/24 19:53

Description

The commitRequestManager calls onFailedAttempt to register a failed request, and this call directly affects the count of attempts, considered to determine the backoff to apply. Therefore, this onFailedAttempts should only be called once per request, but currently the CommitRequestManager may call it multiple times for a single request (if the response contains multiple partition errors):

• on commit: https://github.com/apache/kafka/blob/2f9b23625917fed841a176a73bbbc02bfa330a2d/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L726

• on fetch: https://github.com/apache/kafka/blob/2f9b23625917fed841a176a73bbbc02bfa330a2d/clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java#L1049 

We should ensure that onFailedAttempt is only called once per request, and add a test that covers both cases above, of responses with multiple partition errors in it.

Issue Links

• links to: GitHub Pull Request #17109