[FLINK-25217] FLIP-190: Support Version Upgrades for Table API & SQL Programs

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Table SQL / API, Table SQL / Planner
• Labels: auto-deprioritized-major pull-request-available
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 07/Dec/21 15:49
• Updated: 31/Oct/23 21:47

Description

Nowadays, the Table & SQL API is as important to Flink as the DataStream API. It is one of the main abstractions for expressing pipelines that perform stateful stream processing. Users expect the same backwards compatibility guarantees when upgrading to a newer Flink version as with the DataStream API.

In particular, this means:

• once the operator topology is defined, it remains static and does not change between Flink versions, unless resulting in better performance,
• business logic (defined using expressions and functions in queries) behaves identical as before the version upgrade,
• the state of a Table & SQL API program can be restored from a savepoint of a previous version,
• adding or removing stateful operators should be made possible in the DataStream API.

The same query can remain up and running after upgrades.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489

Issue Links

• incorporates: FLINK-33421 Implement ExecNode Restore Tests
• relates to: FLINK-31884 Upgrade ExecNode to new version causes the old serialized plan failed to pass Json SerDe round trip
• relates to: FLINK-31917 Loss of Idempotence in JsonSerDe Round Trip for AggregateCall and RexNode
• links to: GitHub Pull Request #18925

Sub-Tasks

• Add catalog object compile/restore options: Closed
• Add new STATEMENT SET syntax: Closed
• Support new statement set syntax in sql client and update docs: Closed
• Harden JSON Serialization utilities: Closed
• Harden type serialization in JSON plan: Closed
• Avoid creating temporary tables for inline tables: Closed
• Improve function identifier string in plan digest: Closed
• Make ObjectIdentifier json representation simpler: Closed
• Add internal functions and basic function versioning: Closed
• Handle symbols at a central place with serializable format: Closed
• Properly declare internal UNNEST function: Closed
• Introduce ContextResolvedFunction similar to ContextResolvedTable: Closed
• Harden function serialization in JSON plan: Closed
• Harden AggregateCall serialization in JSON plan: Closed
• Support forward options in DynamicTableFactory: Closed
• Updating existing table factories for mutable table options: Closed
• Use compact DataType serialization for default classes instead of internal ones: Closed
• Harden table serialization in JSON plan: Closed
• Introduce ExecNodeMetadata annotation and tooling: Closed
• Identify topology changing config options for all StreamExec nodes: Closed
• Update JSON plan with new ExecNodeMetadata: Closed
• Use jackson jdk8/time modules for Duration ser/de: Closed
• Introduce ExecNodeContext.generateUid(): Closed
• Expose uid generator for DataStream/Transformation providers: Closed
• LogicalRelDataTypeConverterTest is flaky on Azure: Closed
• Introduce test infra for building FLIP-190 tests: Closed
• Add ExecNodeGraph ser/de: Closed
• Expose plan via TableEnvironment.compilePlanSql/executePlan: Closed
• Expose plan via StatementSet.compilePlan: Closed
• Expose plan via SQL COMPILE / EXECUTE PLAN: Closed
• Introduce TablePipeline: Closed
• Expose plan via Table.compilePlan/TableEnvironment.fromPlan: Open
• Add FLIP-190 new API methods to python: In Progress
• Use new FlinkVersion enum for serde plans: Closed
• Make Python specific exec nodes unsupported: Closed
• TableEnvironment#compilePlan should fail: Closed
• Persist per-ExecNode configuration: Closed
• CompiledPlan should implement Executable: Closed
• Always serialize anonymous table: Closed
• Fix parser generator warnings: Closed
• Document the upgrade story: In Progress
• Add a flag to disable uid generation: Closed
• Introduce ExecutionNodeConfig which will be used in all Exec Nodes: Closed
• Replace TableConfig with PlannerConfig in PlannerContext: Closed
• Improve reporting serialisation issues: Closed
• Make TestValues sinks return results as Rows: Closed
• Add a RestoreTestBase: Resolved
• Avoid CompiledPlan recompilation during loading: Closed
• Add a test to verify if restore test exists for ExecNode: Closed