[FLINK-33724] Application mode doesn't support multiple lines argument

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.18.0
• Component/s: Deployment / YARN
• Assignee: Unassigned
• Reporter: Jiaxing Chen
• Created: 01/Dec/23 10:51
• Updated: 06/Dec/23 01:59
• Resolved: 06/Dec/23 01:59

Description

When running a job in application mode, flink will save program arguments in flink-conf.yaml. These arguments will be loaded when deploying an application on YARN. However multiple lines argument cannot be loaded properly.

In my case, I submit a job with sql which is a multiple lines argument:

 

/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c 
FROM tableB;"

In flink-conf.yaml，it saved as:

$internal.application.program-args： "INSERT INTO tableA 
SELECT 
a,
b,
c 
FROM tableB;"

And produced some warning logs when loaded from flink-conf.yaml :

INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: $internal.application.program-args, "INSERT INTO tableA SELECT a,
WARN  org.apache.flink.configuration.GlobalConfiguration           [] - Error while trying to split key and value in configuration file /{working dir}/flink-conf.yaml:{line num} "SELECT "
...

I dig into the source code and find out that the reader and writer methods of YAML file cannot properly parse YAML format. Maybe it would be better to use some third-party tools, such as snakeyaml, to do this job.