[KAFKA-9678] Introduce bounded exponential backoff in clients

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: admin, consumer, producer
• Labels: needs-kip
• Assignee: Sanjana Kaundinya
• Reporter: Guozhang Wang
• Created: 07/Mar/20 00:53
• Updated: 01/Jul/20 22:10
• Resolved: 01/Jul/20 22:09

Description

In all clients (consumer, producer, admin, and streams) we have retry mechanisms with fixed backoff to handle transient connection issues with brokers. However, with small backoff (many defaults to 100ms) we could send 10s of requests per second to the broker, and if the connection issue is prolonged it means a huge overhead.

We should consider introducing upper-bounded exponential backoff universally in those clients to reduce the num of retry requests during the period of connection partitioning.

Issue Links

• duplicates: KAFKA-9800 [KIP-580] Client Exponential Backoff Implementation
• relates to: KAFKA-9621 AdminClient listOffsets operation does not respect retries and backoff
• relates to: KAFKA-9047 AdminClient group operations may not respect backoff
• relates to: KAFKA-9145 AbstractCoordinator should respect retry backoff between rebalances