[FLINK-25217] FLIP-190: Support Version Upgrades for Table API & SQL Programs

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Table SQL / API, Table SQL / Planner
• Labels: auto-deprioritized-major pull-request-available
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 07/Dec/21 15:49
• Updated: 31/Oct/23 21:47

Description

Nowadays, the Table & SQL API is as important to Flink as the DataStream API. It is one of the main abstractions for expressing pipelines that perform stateful stream processing. Users expect the same backwards compatibility guarantees when upgrading to a newer Flink version as with the DataStream API.

In particular, this means:

• once the operator topology is defined, it remains static and does not change between Flink versions, unless resulting in better performance,
• business logic (defined using expressions and functions in queries) behaves identical as before the version upgrade,
• the state of a Table & SQL API program can be restored from a savepoint of a previous version,
• adding or removing stateful operators should be made possible in the DataStream API.

The same query can remain up and running after upgrades.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489

Issue Links

• incorporates: FLINK-33421 Implement ExecNode Restore Tests
• relates to: FLINK-31884 Upgrade ExecNode to new version causes the old serialized plan failed to pass Json SerDe round trip
• relates to: FLINK-31917 Loss of Idempotence in JsonSerDe Round Trip for AggregateCall and RexNode
• links to: GitHub Pull Request #18925

Sub-Tasks

• Expose plan via Table.compilePlan/TableEnvironment.fromPlan: Open
• Add FLIP-190 new API methods to python: In Progress
• Document the upgrade story: In Progress