[KAFKA-5827] Allow configuring Kafka sink connectors to start processing records from the end of topics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: connect
• Assignee: Unassigned
• Reporter: Behrang Saeedzadeh
• Created: 03/Sep/17 14:10
• Updated: 23/Feb/23 16:09
• Resolved: 23/Feb/23 16:09

Description

As far as I can see, Kafka connectors start exporting data of a topic from the beginning of its partitions. We have a topic that contains a few million old records that we don't need but we would like to start exporting new records that are added to the topic.

Basically:

• When the connector is started for the first time and it does not have a current offset stored, it should start consuming data from the end of topic partitions
• When the connector is restarted and has a current offset for partitions stored somewhere, it should start from those offsets

Issue Links

• is fixed by: KAFKA-8265 Connect Client Config Override policy