[FLINK-23620] Introduce proper YAML parsing to Flink's configuration

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / Configuration
• Labels: 2.0-related
• Assignee: Junrui Li
• Reporter: Mika Naylor
• Created: 04/Aug/21 12:04
• Updated: 01/Dec/23 03:27

Description

At the moment, the YAML parsing for Flink's configuration file (conf/flink-conf.yaml) is pretty basic. It only supports basic key value pairs, such as:

a.b.c: a value
a.b.d: another value

As well as supporting some invalid YAML syntax, such as:

a: b: value

 

Introducing proper YAML parsing to the configuration component would let Flink users use features such as nested keys, such as:

a:
  b:
    c: a value
    d: another value

as well as make it easier to integrate configuration tools/languages that compile to YAML, such as Dhall.

Issue Links

• is duplicated by: FLINK-27299 flink parsing parameter bug
• is duplicated by: FLINK-29366 Use flink-shaded-jacson library to parse flink-conf.yaml
• is duplicated by: FLINK-27221 Improvements to YAML parsing
• is related to: FLINK-33297 FLIP-366: Support standard YAML for FLINK configuration
• relates to: FLINK-15358 [configuration] the global configuration will trim the rest of value after a `#` comment sign
• relates to: FLINK-32383 2.0 Breaking configuration changes
• links to: GitHub Pull Request #17821

Sub-Tasks

• Add a suffix to keys that violate YAML spec: Closed
• Add checks to guarantee the non-deprecated options not conflicting with standard YAML: Closed