Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
Description
Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and PRIMARY_WORKER-launch-script****.sh to staging dir.
The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job is submitted multiple times.
But PRIMARY_WORKER-launch-script****.sh would not be overwritten, as it has random numbers in its name.
The files in the staging dir are as follows:
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:11 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:02 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:06 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh -rw-r----- 2 hadoop hdfs 15225 2018-08-17 18:46 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh -rw-r----- 2 hadoop hdfs 580 2018-08-16 20:48 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh -rw-r----- 2 hadoop hdfs 580 2018-08-17 14:53 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:16 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh -rw-r----- 2 hadoop hdfs 8815 2018-08-27 15:54 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml -rw-r----- 2 hadoop hdfs 11583 2018-08-27 15:54 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56 hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
We should stop the staging dir from growing or have a way to clean it up