[SPARK-4461] Pass java options to yarn master to handle system properties correctly.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: YARN
• Target Version/s: 1.3.0
• Assignee: Zhan Zhang
• Reporter: Zhan Zhang
• Created: 17/Nov/14 22:28
• Updated: 18/Dec/14 16:02
• Resolved: 18/Dec/14 16:02

Description

Currently spark read mapred-site.xml to get the class path. From hadoop 2.6, the library is shipped to cluster with distributed cache at run-time, and may not be available at every node manager.

Instead of relying on mapred-site.xml, spark should handle this by its own, for example, through ADD_JARs, SPARK_CLASSPATH, etc

Issue Links

• is related to: SPARK-4181 Create separate options to control the client-mode AM resource allocation request
• links to: [Github] Pull Request #3409 (zhzhan)