[SPARK-8617] Handle history files better

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Ergin Seyfe
• Reporter: Thomas Graves
• Created: 25/Jun/15 02:14
• Updated: 15/Dec/16 21:05
• Resolved: 08/Dec/16 18:21

Description

Right now there are many conditions in which history files get left around with the .inprogress extension. The cleaner doesn't remove these because it can't distinguish between something running and left over abandoned files.

There are cases its really hard to not leave these around (application killed/crashes/etc). But we should still try to handle the history files better.

For instance we could first write them to the .sparkStaging directory and then moving to the history file location before the application finishes. That way you only get completed history files.

Issue Links

• is duplicated by: SPARK-18733 Spark history server file cleaner excludes in-progress files
• is duplicated by: SPARK-17119 Add configuration property to allow the history server to delete .inprogress files
• links to: [Github] Pull Request #16165 (seyfe)