[FLINK-28303] Kafka SQL Connector loses data when restoring from a savepoint with a topic with empty partitions

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.14.4, 1.15.4, kafka-3.0.0, 1.16.2, 1.17.1
• Fix Version/s: kafka-3.0.1, kafka-3.1.0
• Component/s: Connectors / Kafka
• Labels: pull-request-available
• Assignee: tanjialiang
• Reporter: Robert Metzger
• Created: 29/Jun/22 17:27
• Updated: 24/Oct/23 18:14
• Resolved: 23/Oct/23 18:30

Description

Steps to reproduce:

• Set up a Kafka topic with 10 partitions
• produce records 0-9 into the topic
• take a savepoint and stop the job
• produce records 10-19 into the topic
• restore the job from the savepoint.

The job will be missing usually 2-4 records from 10-19.

My assumption is that if a partition never had data (which is likely with 10 partitions and 10 records), the savepoint will only contain offsets for partitions with data.
While the job was offline (and we've written record 10-19 into the topic), all partitions got filled. Now, when Kafka comes online again, it will use the "latest" offset for those partitions, skipping some data.

Issue Links

• links to: GitHub Pull Request #52