[KAFKA-12361] Change default connect producer request timeout

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: connect
• Assignee: Chris Egerton
• Reporter: Jason Gustafson
• Created: 22/Feb/21 19:04
• Updated: 24/Feb/21 02:32
• Resolved: 24/Feb/21 02:32

Description

Prior to KIP-91, which introduced delivery.timeout.ms, there was no easy way to ensure that records sent through the producer would even have the opportunity to get delivered. Records could be timed out in the accumulator if `request.timeout.ms` was reached before getting sent. Users worked around this problem by setting `request.timeout.ms=Int.MaxValue`. The downside is that this made the producer slower to discover "unclean" connection failures. Now that we have KIP-91, there shouldn't be any reason to keep this workaround.

One place it would be good to fix this is in connect's source tasks:

        // These settings will execute infinite retries on retriable exceptions. They *may* be overridden via configs passed to the worker,
        // but this may compromise the delivery guarantees of Kafka Connect.
        producerProps.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, Integer.toString(Integer.MAX_VALUE));

The comment about delivery guarantees is a little vague, but I think mainly it is what was discussed above about ensuring at least once delivery. Note that none of the default configs including both request timeout and delivery timeout can avoid duplicates in all cases. For that idempotence is needed. It is worth considering separately for connect whether that should be the default.

Issue Links

• links to: GitHub Pull Request #10178