[SPARK-18733] Spark history server file cleaner excludes in-progress files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.2
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: Ergin Seyfe
• Created: 06/Dec/16 04:53
• Updated: 06/Dec/16 21:03
• Resolved: 06/Dec/16 20:39

Description

When we restart history server, it does spend a lot of time to load/replay incomplete applications which mean the inprogress log files in the log folder.

We have already enabled "spark.history.fs.cleaner.enabled" but seems like it's skipping the inprogress files.

I checked the log folder and saw that there are many old orphan files. Probably files left over due to spark-driver failures or OOMs.

Issue Links

• duplicates: SPARK-8617 Handle history files better
• links to: [Github] Pull Request #16165 (seyfe)