[FLINK-10806] Support multiple consuming offsets when discovering a new topic

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.6.2, 1.8.1
• Component/s: Connectors / Kafka
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Jiayi Liao
• Created: 07/Nov/18 03:40
• Updated: 26/Dec/21 10:39

Description

In KafkaConsumerBase, we discover the TopicPartitions and compare them with the restoredState. It's reasonable when a topic's partitions scaled. However, if we add a new topic which has too much data and restore the Flink program, the data of the new topic will be consumed from the start, which may not be what we want. I think this should be an option for developers.

Issue Links

• is duplicated by: FLINK-16865 【Flink Kafka Connector】Restore from Savepoint，if add new Kafka topic，Flink will consume the new topic from earlist，it may cause duplicate data sink