[FLINK-17412] Make the FLINK_YARN_FILES value configurable

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.7.3, 1.8.0
• Component/s: Client / Job Submission, Deployment / YARN
• Assignee: Unassigned
• Reporter: opencoder
• Created: 27/Apr/20 10:28
• Updated: 13/Apr/21 20:30
• Resolved: 23/Oct/20 09:47

Description

When the application is deployed to YARN, the file of the application will be uploaded to the directory returned by

AbstractYarnClusterDescriptor#getYarnFilesDir(ApplicationId) 

method. When the same user name is used to submit applications to different clusters, it may cause directory name conflicts(because different YARN clusters may generate the same applicationId). In addition, when the application is directly killed by the user, the corresponding directory will continue to exist and occupy disk space, and the judgment conditions for cleaning such invalid directories are complex. Therefore, it is hoped that a configuration option can be added to support each application to dynamically specify the directory where its files are located during deployment.

Issue Links

• duplicates: FLINK-18493 Make target home directory used to store yarn files configurable
• is related to: FLINK-17472 StreamExecutionEnvironment and ExecutionEnvironment in Yarn mode