[FLINK-26075] Persist per-ExecNode configuration

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Table SQL / Planner
• Labels: pull-request-available
• Assignee: Marios Trivyzas
• Reporter: Timo Walther
• Created: 10/Feb/22 14:15
• Updated: 26/Mar/22 06:25
• Resolved: 26/Mar/22 06:25

Description

Even though a compiled plan is static, some configuration options still change the topology of an ExecNode. In general, we will request users to keep the configuration constant between Flink versions. However, setting configuration more fine-grained per-ExecNode is a frequently requested feature. It can also allow us to set the parallelism more fine-grained in the future.

We need the following infrastructure for the mentioned use cases:

• Every ExecNode can have a configuration
• By default the configuration per node are the values from the global configuration using the keys from the ExecNodeMetadata annotation.
• We persist the ExecNode configuration in the JSON plan.
• If the persisted plan contains a configuration, the persisted configuration is merged with the global configuration and has priority.

Issue Links

• Dependency: FLINK-26297 Introduce ExecutionNodeConfig which will be used in all Exec Nodes
• links to: GitHub Pull Request #18756
• links to: GitHub Pull Request #18908
• links to: GitHub Pull Request #19232
• links to: GitHub Pull Request #19247