[FLINK-32019] EARLIEST offset strategy for partitions discoveried later based on FLIP-288

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: kafka-3.0.0
• Fix Version/s: kafka-3.1.0
• Component/s: Connectors / Kafka
• Labels: pull-request-available
• Assignee: Hongshun Wang
• Reporter: Hongshun Wang
• Created: 06/May/23 02:50
• Updated: 26/Jan/24 08:33
• Resolved: 07/Jul/23 02:21

Description

As described in FLIP-288, the strategy used for new partitions is the same as the initial offset strategy, which is not reasonable.

According to the semantics, if the startup strategy is latest, the consumed data should include all data from the moment of startup, which also includes all messages from new created partitions. However, the latest strategy currently maybe used for new partitions, leading to the loss of some data (thinking a new partition is created and might be discovered by Kafka source several minutes later, and the message produced into the partition within the gap might be dropped if we use for example "latest" as the initial offset strategy).if the data from all new partitions is not read, it does not meet the user's expectations.

Other ploblems see final Section of FLIP-288: User specifies OffsetsInitializer for new partition .

Therefore, it’s better to provide an EARLIEST strategy for later discovered partitions.

Issue Links

• links to: GitHub Pull Request #28