[FLINK-33980] Reorganize job configuration

• Type: Technical Debt
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.19.0
• Component/s: API / Core
• Labels: pull-request-available
• Assignee: Junrui Li
• Reporter: Junrui Li
• Created: 04/Jan/24 03:52
• Updated: 16/Jan/24 06:53
• Resolved: 16/Jan/24 06:53

Description

Currently, job configuration in FLINK is spread out across different components, including StreamExecutionEnvironment, CheckpointConfig, and ExecutionConfig. This distribution leads to inconsistencies among the configurations stored within these components. Furthermore, the methods used to configure these components vary; some rely on complex Java objects, while others use ConfigOption, which is a key-value configuration approach. This variation complicates the effective management of job configurations. Additionally, passing complex Java objects (e.g., StateBackend and CheckpointStorage) between the environment, StreamGraph, and JobGraph adds complexity to development.

With the completion of FLIP-381, it is now time to standardize and unify job configuration in FLINK. The goals of this JIRA are as follows:

1. Migrate configuration from non-ConfigOption objects to use ConfigOption.
2. Adopt a single Configuration object to house all configurations.
3. Create complex Java objects, such as RestartBackoffTimeStrategyFactory, CheckpointStorage, and StateBackend, directly from the configuration on the JM side.

Issue Links

• relates to: FLINK-33581 FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects
• links to: GitHub Pull Request #24025