Description
There's currently an incompatibility between Sink connectors overriding the SinkTask.preCommit method (for asynchronous processing) and SMTs that mutate the topic field.
The problem was present since the preCommit method inception and is rooted in a mismatch between the topic/partition that is passed to open/preCommit (the original topic and partition before applying any transformations) and the topic partition that is present in the SinkRecord that the SinkTask.put method receives (after transformations are applied). Since that's all the information the connector has to implement any kind of internal offset tracking, the topic/partitions it can return in preCommit will correspond to the transformed topic, when the framework actually expects it to be the original topic.
Attachments
Issue Links
- is caused by
-
KAFKA-5568 Transformations that mutate topic-partitions break sink connectors that manage their own configuration
-
- Open
-
- links to
- mentioned in
-
Page Loading...