[FLINK-15447] To improve utilization of the `java.io.tmpdir` for YARN module

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.9.1
• Component/s: Deployment / YARN
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: jiasheng55
• Created: 31/Dec/19 09:58
• Updated: 20/Mar/20 03:55
• Resolved: 20/Mar/20 03:55

Description

#Background

Currently, when running Flink on Yarn, the "java.io.tmpdir" property is set to the default value, which is "/tmp".  

Sometimes we ran into exceptions caused by a full "/tmp" directory, which would not be cleaned automatically after applications finished.

 

#Goal

quoted from: HADOOP-2735

1) Tasks can utilize all disks when using tmp
2) Any undeleted tmp files will be deleted by the tasktracker when task(job?) is done.

 

#Suggestion

I think we can set "java.io.tmpdir" to "{{PWD}}/tmp" directory, or something similar. "{{PWD}}" will be replaced with the true working directory of JM/TM by Yarn, which will be cleaned automatically.

 

Issue Links

• links to: GitHub Pull Request #11331