[SPARK-26877] Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Assignee: liupengcheng
• Reporter: liupengcheng
• Created: 14/Feb/19 07:57
• Updated: 17/May/20 18:14
• Resolved: 20/Feb/19 19:45

Description

Currently, when running applications on yarn mode, the app staging directory of  is controlled by `spark.yarn.stagingDir` config if specified, and this directory cannot separate different users, somtimes, it's inconvenient for file and quota management for users.

For example, user may have different quota for their own app staging dir. and they may also not want to use the home directory as the staging dir, because it might used mainly for private user files or data. The quota may also be different between private data and the app staging dir.

So I propose to add user sub directories under this app staging dir.

Issue Links

• links to: GitHub Pull Request #23786