[KAFKA-5567] With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10.2.0, 0.10.2.1, 0.11.0.0
• Fix Version/s: 0.11.0.1, 1.0.0
• Component/s: KafkaConnect
• Assignee: Konstantine Karantasis
• Reporter: Konstantine Karantasis
• Reviewer: Ewen Cheslack-Postava
• Created: 07/Jul/17 00:55
• Updated: 24/Dec/18 12:37
• Resolved: 16/Aug/17 21:44

Description

When a chain of transformations (SMTs) that mutate a record's topic-partition is applied, then Connect is unable to map the transformed record to its original topic-partition. This affects committing offsets.

Currently, in order to reproduce the issue one could use the TimestampRouter transformation with a sink connector such as the FileStreamSinkConnector.

In this ticket we'll address the issue for connectors that don't manage/commit their offsets themselves. For the connectors that do such management, broader API changes are required to supply the connectors with the necessary information that will allow them to map a transformed record to the original.

Issue Links

• links to: GitHub Pull Request #3499