[FLINK-15470] Remove YARN magic properties file

• Type: Sub-task
• Status: Reopened
• Priority: Not a Priority
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Command Line Client, Deployment / YARN
• Assignee: Unassigned
• Reporter: Yang Wang
• Created: 03/Jan/20 11:17
• Updated: 29/Jul/21 07:37

Description

Currently, when we start a session, flink client will write a yarn properties under {java.io.tmpdir}/.yarn-properties-{user-name}. So the user could use a simple command ./bin/flink run ./examples/batch/WordCount.jar to submit a job to the session.

 

Even if the session cluster has finished or kill by yarn application -kill, it will still submit a job to the non-existing session. Some users has run into this problem in the ML. So i suggest to remove the properties file and always use -yid application_id to submit job an existing session.

https://lists.apache.org/thread.html/b0e0e388a16d1891d1de651bec027a3820130f28aaaeb843440cb679%40%3Cuser.flink.apache.org%3E

 

We also have some discussion under FLINK-15179. The cli options will be deprecated in the future and we hope FlinkYarnSessionCli could use ExecutorCli. So we will try to clean up the code, i think the magic properties file is the case.

 

cc tison kkl0u

Issue Links

• relates to: FLINK-15437 Start session with property of "-Dtaskmanager.memory.process.size" not work