[FLINK-16305] FlinkYarnSessionClI ignores target executor and uses yarn-session if YARN properties file is present

• Type: Bug
• Status: In Progress
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: Client / Job Submission, Command Line Client
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Daniel Laszlo Magyar
• Created: 27/Feb/20 10:18
• Updated: 17/Dec/21 22:38

Description

The presence of the hidden YARN property file (which contains the last started YARN session’s application id), causes the cli to ignore the execution.target property set in the conf/flink-conf.yaml configuration file, which leads to unexpected behaviour at the time of job submission via cli, e.g. when using flink run or SQL client.
The code that ignores the execution target if the YARN application id is set in the hidden property file is at https://github.com/apache/flink/blob/release-1.10/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L337-L351.

Reproduction steps:
• start flink yarn session via ./bin/yarn-session.sh -d, this writes the application id to /tmp/.yarn-properties-root
• set execution.target: yarn-per-job in /etc/flink/conf/flink-conf.yaml
• enable debug logging
• run a flink job e.g. flink run -d -p 2 examples/streaming/WordCount.jar --input README.txt
• the logs below show that even though the execution.target property is read properly, FlinkYarnSessionCli is chosen and the execution.target is reset to yarn-session

20/02/26 12:14:24 INFO configuration.GlobalConfiguration: Loading configuration property: execution.target, yarn-per-job
...
20/02/26 12:14:24 INFO cli.FlinkYarnSessionCli: Found Yarn properties file under /tmp/.yarn-properties-root.
20/02/26 12:14:24 DEBUG fs.FileSystem: Loading extension file systems via services
20/02/26 12:14:24 DEBUG cli.CliFrontend: Custom commandlines: [org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, org.apache.flink.client.cli.ExecutorCLI@6d60fe40, org.apache.flink.client.cli.DefaultCLI@792b749c]
20/02/26 12:14:24 DEBUG cli.CliFrontend: Checking custom commandline org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, isActive: true
...
20/02/26 12:14:25 DEBUG cli.CliFrontend: Effective executor configuration: {...execution.target=yarn-session, ....}
20/02/26 12:14:25 INFO client.ClientUtils: Starting program (detached: true)