[SUBMARINE-18] Enable local staging directory and clean it up when submarine job is submitted - ASF JIRA

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Won't Fix
Affects Version/s: None
Fix Version/s: None
Component/s: Commons
Labels:
None
Environment:

Hide
In the /tmp dir, there are launch scripts which are not cleaned up as follows:

-rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 PRIMARY_WORKER-launch-script8635233314077649086.sh
-rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 WORKER-launch-script129488020578466938.sh
-rw-r--r-- 1 hadoop netease 1028 Sep 18 10:46 PS-launch-script471092031021738136.sh

Show
In the /tmp dir, there are launch scripts which are not cleaned up as follows: -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 PRIMARY_WORKER-launch-script8635233314077649086.sh -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 WORKER-launch-script129488020578466938.sh -rw-r--r-- 1 hadoop netease 1028 Sep 18 10:46 PS-launch-script471092031021738136.sh

Description

YarnServiceJobSubmitter.generateCommandLaunchScript creates container launch scripts in the local filesystem. Container launch scripts would be uploaded to hdfs staging dir, but would not be not deleted after the job is submitted

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

YARN-8806.006.patch
26/Sep/18 05:06
10 kB
Zac Zhou
YARN-8806.005.patch
25/Sep/18 07:35
10 kB
Zac Zhou
YARN-8806.004.patch
25/Sep/18 06:43
11 kB
Zac Zhou
YARN-8806.003.patch
21/Sep/18 07:53
14 kB
Zac Zhou
YARN-8806.002.patch
20/Sep/18 15:22
8 kB
Zac Zhou
YARN-8806.001.patch
20/Sep/18 14:43
9 kB
Zac Zhou

Activity

People

Assignee:: Zac Zhou

Reporter:: Zac Zhou

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 20/Sep/18 14:40

Updated:: 26/Jun/20 14:57

Resolved:: 10/Mar/20 02:56