[FLINK-32548] Make watermark alignment ready for production use

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.2, 1.17.1
• Fix Version/s: 1.18.0, 1.16.3, 1.17.2

Release Note

The watermark alignment is ready for production since Flink-1.18, which completed a series of bug fixes and improvements releted to watermark alignment. As proven by the micro benchmarks (screenshots attached in FLINK-32420), with 5000 subtasks, the time to calculate the watermark alignment on the JobManager by a factor of 76x (7664%). Previously such large jobs where actually at large risk of overloading JobManager, now that's far less likely to happen.

• Assignee: Rui Fan
• Reporter: Rui Fan
• Created: 06/Jul/23 06:53
• Updated: 02/Aug/23 02:11
• Resolved: 17/Jul/23 07:35

Description

We found a series of watermark alignment bugs and performance issues and hope to reach production availability in 1.18.0.

And fixes all bugs found in 1.16.3 and 1.17.2.

Issue Links

• is a parent of: FLINK-32524 Improve the watermark aggregation performance when enabling the watermark alignment

Sub-Tasks

• Duplicated announceCombinedWatermark task maybe scheduled if jobmanager failover: Closed
• SourceAlignment announceCombinedWatermark period task maybe lost: Closed
• SourceCoordinator thread leaks when job recovers from checkpoint: Resolved
• Watermark alignment will cause flink jobs to hang forever when any source subtask has no SourceSplit: Resolved
• Sources with idleness and alignment always wait for alignment when part of multiple sources is idle: Resolved
• Remove the beta for watermark alignment: Resolved