[KAFKA-13348] Allow Source Tasks to Handle Producer Exceptions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 2.8.1, 3.0.0
• Fix Version/s: 3.2.0
• Component/s: connect
• Assignee: Knowles Atchison Jr
• Reporter: Knowles Atchison Jr
• Created: 05/Oct/21 16:58
• Updated: 27/Jan/22 18:20
• Resolved: 27/Jan/22 18:20

Description

KAFKA-8586 added capture of Producer Exceptions which will kill the connector.

There is a need to allow the connector itself to be aware of these errors, handle it in some manner, and continuing processing records.

The proposed change is to add a getter to the retry operator and allow producer failure to be ignored via the "errors.tolerance" configuration setting. The connector will receive a callback to commitRecord() with a null MetadataRecord.

The current behavior is kept if errors.tolerance is not set to all.

 

see: https://cwiki.apache.org/confluence/display/KAFKA/KIP-779%3A+Allow+Source+Tasks+to+Handle+Producer+Exceptions

Issue Links

• links to: GitHub Pull Request #11382