[BEAM-6733] Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink
• Assignee: Unassigned
• Reporter: Maximilian Michels
• Created: 22/Feb/19 17:24
• Updated: 03/Jun/22 23:47

Description

Flink 1.6/1.7 provides a hook to execute an action before the snapshot barrier is emitted by the operator. At the moment (<=1.5) the Flink Runner has to buffer any elements which are emitted during a snapshot because the barrier has already been emitted. This leads to a lot of code complexity.

We can remove the buffering in favor of finishing the current bundle in DoFnOperator's prepareSnapshotPreBarrier. The 1.5/1.6/1.7 build setup poses a challenge to do that in a way that does not lead to much code duplication.

Issue Links

• is related to: BEAM-7962 Drop support for Flink 1.5 and 1.6
• links to: GitHub Pull Request #7940
• links to: GitHub Pull Request #9652
• links to: GitHub Pull Request #11678