[SPARK-1865] Improve behavior of cleanup of disk state

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Deploy, Spark Core
• Assignee: Unassigned
• Reporter: Aaron Davidson
• Created: 17/May/14 03:09
• Updated: 08/Oct/16 09:33
• Resolved: 08/Oct/16 09:33

Description

Right now the behavior of disk cleanup is centered around the exit hook of the executor, which attempts to cleanup shuffle files and disk manager blocks, but may fail. We should make this behavior more predictable, perhaps by letting the Standalone Worker cleanup the disk state, and adding a flag to disable having the executor cleanup its own state.

Issue Links

• is related to: SPARK-12330 Mesos coarse executor does not cleanup blockmgr properly on termination if data is stored on disk