[SPARK-11843] Isolate staging directory across applications on same YARN cluster

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Andrew Or
• Created: 19/Nov/15 02:19
• Updated: 17/May/20 18:16
• Resolved: 20/Nov/15 01:43

Description

If multiple clients share the same YARN cluster and file system they may end up using the same `.sparkStaging` directory. This may be a problem if their jars are called something similar, for instance. It would be easier to enforce isolation for both security and user experience if the staging directories are isolated. We can either:

(1) allow users to configure the directory name
(2) add an identifier to the directory name, which I prefer