[SPARK-10975] Shuffle files left behind on Mesos without dynamic allocation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Component/s: Mesos
• Assignee: Unassigned
• Reporter: Dragos Dascalita Haut
• Created: 07/Oct/15 15:39
• Updated: 18/Jan/16 20:46
• Resolved: 04/Nov/15 12:12

Description

(from mailing list)

Running on Mesos in coarse-grained mode. No dynamic allocation or shuffle service.

I see that there are two types of temporary files under /tmp folder associated with every executor: /tmp/spark-<UUID> and /tmp/blockmgr-<UUID>. When job is finished /tmp/spark-<UUID> is gone, but blockmgr directory is left with all gigabytes in it.

The reason is that logic to clean up files is only enabled when the shuffle service is running, see https://github.com/apache/spark/pull/7820

The shuffle files should be placed in the Mesos sandbox or under `tmp/spark` unless the shuffle service is enabled.

Issue Links

• duplicates: SPARK-9708 Spark should create local temporary directories in Mesos sandbox when launched with Mesos
• relates to: SPARK-9708 Spark should create local temporary directories in Mesos sandbox when launched with Mesos