[FLINK-34114] Parse error while $internal.application.program-args contains '#' in Yarn/K8s Application Mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.18.0
• Component/s: Deployment / Kubernetes, Deployment / YARN
• Assignee: Unassigned
• Reporter: jonasjc
• Created: 16/Jan/24 11:06
• Updated: 17/Jan/24 02:44
• Resolved: 17/Jan/24 02:44

Description

When run job by K8s or Yarn Application Mode use org.apache.flink.configuration.GlobalConfiguration#loadYAMLResource method parse config。

 

private static Configuration loadYAMLResource(File file) {
    final Configuration config = new Configuration();

    try (BufferedReader reader =
            new BufferedReader(new InputStreamReader(new FileInputStream(file)))) {

        String line;
        int lineNo = 0;
        while ((line = reader.readLine()) != null) {
            lineNo++;
            // 1. check for comments
            String[] comments = line.split("#", 2);
            String conf = comments[0].trim();

            // 2. get key and value
            if (conf.length() > 0) {
                String[] kv = conf.split(": ", 2);

                // skip line with no valid key-value pair
                if (kv.length == 1) {
                    LOG.warn(
                            "Error while trying to split key and value in configuration file "
                                    + file
                                    + ":"
                                    + lineNo
                                    + ": Line is not a key-value pair (missing space after ':'?)");
                    continue;
                }

                String key = kv[0].trim();
                String value = kv[1].trim();

                // sanity check
                if (key.length() == 0 || value.length() == 0) {
                    LOG.warn(
                            "Error after splitting key and value in configuration file "
                                    + file
                                    + ":"
                                    + lineNo
                                    + ": Key or value was empty");
                    continue;
                }

                config.setString(key, value);
            }
        }
    } catch (IOException e) {
        throw new RuntimeException("Error parsing YAML configuration.", e);
    }

    return config;
} 

if config value contains '#' like

 

$internal.application.program-args: '{test:test#jsonstring}'

 

 

the following code is not correct

line.split("#", 2) 

To fix this,i think we should import snakeyaml to fix complex situation

Issue Links

• Blocked: FLINK-33297 FLIP-366: Support standard YAML for FLINK configuration