[FLINK-26254] KafkaSink might violate order of sequence numbers and risk exactly-once processing

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.14.3, 1.15.0
• Component/s: Connectors / Kafka
• Assignee: Alexander Preuss
• Reporter: Fabian Paul
• Created: 18/Feb/22 14:20
• Updated: 23/Nov/22 08:24
• Resolved: 23/Nov/22 08:24

Description

When running the KafkaSink in exactly-once mode with a very low checkpoint interval users are seeing `OutOfOrderSequenceException`.

It could be caused by the fact that the connector has a pool of KafkaProducers and the sequence numbers are not shared/reset if a new KafkaProducer tries to write to a partition while the previous KafkaProducer is still occupied for committing.