[FLINK-35630] Kafka source may reset the consume offset to earliest when the partition leader changes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: kafka-3.2.0
• Component/s: Connectors / Kafka
• Assignee: Unassigned
• Reporter: tanjialiang
• Created: 18/Jun/24 06:54
• Updated: 20/Jun/24 04:07

Description

Kafka producer using the ack=1 option to write data to a topic.

Flink Kafka source startup with the scan.startup.mode=earliest-offset option to consume (the Kafka auto.offset.reset option will be force override to earliest).

If a partition leader is not available, a follower may become the new leader and this may trigger log truncation. It may cause consumers to consume offset out of range and use the auto.offset.reset strategy to reset the offset.