[SPARK-20741] SparkSubmit does not clean up after uploading spark_libs to the distributed cache

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.2.0
• Component/s: Spark Submit
• Assignee: Lior Regev
• Reporter: Lior Regev
• Created: 15/May/17 08:52
• Updated: 25/May/17 16:09
• Resolved: 25/May/17 16:08

Description

Running spark submit has to distribute Spark's JARs to a distributed cache in order for the executors to access it.
When neither spark.yarn.jars or spark.yarn.archive is provided, SparkSubmit creates a ZIP of all the JARs in $SPARK_HOME/jars and uploads it to the distributed cache.
After uploading the ZIP file, SparkSubmit does not delete the local copy of it. This, in turn can cause the disk on the local machine to fill up (200MB at a time) until no more submissions are possible.

Issue Links

• links to: [Github] Pull Request #17986 (liorregev)