[SPARK-6469] Improving documentation on YARN local directories usage

• Type: Documentation
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.1, 1.4.0
• Component/s: Documentation, YARN
• Target Version/s: 1.3.1, 1.4.0
• Assignee: Christophe Préaud
• Reporter: Christophe Préaud
• Created: 23/Mar/15 14:04
• Updated: 25/Mar/15 00:07
• Resolved: 25/Mar/15 00:07

Description

According to the Spark YARN doc page, Spark executors will use the local directories configured for YARN, not spark.local.dir which should be ignored.

However it should be noted that in yarn-client mode, though the executors will indeed use the local directories configured for YARN, the driver will not, because it is not running on the YARN cluster; the driver in yarn-client will use the local directories defined in spark.local.dir

Can this please be clarified in the Spark YARN documentation above?

Issue Links

• links to: [Github] Pull Request #5165 (preaudc)