[FLINK-28853] FLIP-217 Support watermark alignment of source splits

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0
• Component/s: Connectors / Common, Runtime / Checkpointing
• Labels: pull-request-available

Release Note

Added support for watermark alignment of source splits.

Since Flink 1.17, source connectors have to implement watermark alignment of source split in order to use the watermark alignment feature. The required methods to implement are: `SourceReader#pauseOrResumeSplits` and `SplitReader#pauseOrResumeSplits`. If you are migrating from Flink <= 1.16.x, and you were using watermark alignment, but at the same time you are not able to upgrade/modify your connector, you can disable per split alignment via setting `pipeline.watermark-alignment.allow-unaligned-source-splits` to true. Note that by doing so, watermark alignment will be working properly only when your number of splits equals to the parallelism of the source operator.

• Assignee: Maximilian Michels
• Reporter: Sebastian Mattheis
• Created: 07/Aug/22 17:53
• Updated: 06/Jun/23 08:28
• Resolved: 14/Sep/22 12:58

Description

This improvement implements FLIP-217 to support watermark alignment of source splits.

Issue Links

• causes: FLINK-31324 Broken SingleThreadFetcherManager constructor API
• duplicates: FLINK-27117 FLIP-217: Support watermark alignment of source splits
• is related to: FLINK-27117 FLIP-217: Support watermark alignment of source splits
• relates to: FLINK-12675 Event time synchronization in Kafka consumer
• relates to: FLINK-31519 The watermark alignment docs is outdated after FLIP-217 finished
• links to: GitHub Pull Request #20485
• links to: GitHub Pull Request #22531
• links to: GitHub Pull Request #22716