[SPARK-18968] .sparkStaging quickly fill up HDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.2
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Chen He
• Created: 21/Dec/16 23:18
• Updated: 22/Dec/16 09:52
• Resolved: 22/Dec/16 09:52

Description

We are running streaming jobs using spark. Even the "spark.yarn.preserve.staging.files" is set to b "false", HDFS is quickly been filled up.

Also find people ask mail-list similar question but no further response. http://apache-spark-user-list.1001560.n3.nabble.com/HDFS-folder-sparkStaging-not-deleted-and-filled-up-HDFS-in-yarn-mode-td7851.html

Issue Links

• duplicates: SPARK-18027 .sparkStaging not clean on RM ApplicationNotFoundException