[KAFKA-15415] In Java-client, backoff should be skipped for retried producer-batch to a new leader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.7.0, 3.6.1
• Component/s: clients
• Assignee: Mayank Shekhar Narula
• Reporter: Mayank Shekhar Narula
• Reviewer: Walker Carlson
• Created: 29/Aug/23 10:45
• Updated: 06/Feb/24 15:07
• Resolved: 27/Oct/23 15:31

Description

Right now in java-client, producer-batches backoff upto retry.backoff.ms(100ms by default). This Jira proposes that backoff should be skipped if client knows of a newer-leader for the partition in a sub-sequent retry(typically through refresh of parition-metadata via the Metadata RPC). This would help improve the latency of the produce-request around when partition leadership changes.

Issue Links

• Dependency: KAFKA-15627 KIP-951, Java client changes to incorporate the leader discovery optimisations
• Dependency: KAFKA-15970 KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva
• is a child of: KAFKA-15868 KIP-951 - Leader discovery optimisations for the client
• is cloned by: KAFKA-15824 SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
• is cloned by: KAFKA-16226 Java client: Performance regression in Trogdor benchmark with high partition counts
• links to: GitHub Pull Request #14384
• links to: GitHub Pull Request #14522