[KAFKA-5567] With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: 0.10.2.0, 0.10.2.1, 0.11.0.0
Fix Version/s: 0.11.0.1, 1.0.0
Component/s: KafkaConnect
Labels:
None

Description

When a chain of transformations (SMTs) that mutate a record's topic-partition is applied, then Connect is unable to map the transformed record to its original topic-partition. This affects committing offsets.

Currently, in order to reproduce the issue one could use the TimestampRouter transformation with a sink connector such as the FileStreamSinkConnector.

In this ticket we'll address the issue for connectors that don't manage/commit their offsets themselves. For the connectors that do such management, broader API changes are required to supply the connectors with the necessary information that will allow them to map a transformed record to the original.