[BEAM-5396] Flink portable runner savepoint / upgrade support

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.10.0, 2.11.0
• Component/s: runner-flink
• Labels: portability portability-flink
• Assignee: Maximilian Michels
• Reporter: Thomas Weise
• Created: 17/Sep/18 14:42
• Updated: 16/May/20 17:06
• Resolved: 21/Jan/19 05:43

Description

The portable Flink runner needs to support Flink savepoints for production use. It should be possible to upgrade a stateful portable Beam pipeline that runs on Flink, which involves taking a savepoint and then starting the new version of the pipeline from that savepoint. The potential issues with pipeline evolution and migration are similar to those when using the Flink DataStream API (schema / name changes etc.).

Issue Links

• is related to: BEAM-4289 How to express "Job Update" in portable Beam
• is related to: FLINK-11048 Ability to programmatically execute streaming pipeline with savepoint restore
• links to: GitHub Pull Request #7169
• links to: GitHub Pull Request #7362
• links to: GitHub Pull Request #7584