[SPARK-6797] Add support for YARN cluster mode

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: SparkR
• Target Version/s: 1.5.0
• Assignee: Sun Rui
• Reporter: Shivaram Venkataraman
• Created: 09/Apr/15 05:48
• Updated: 17/Oct/16 02:08
• Resolved: 13/Jul/15 15:22

Description

SparkR currently does not work in YARN cluster mode as the R package is not shipped along with the assembly jar to the YARN AM. We could try to use the support for archives in YARN to send out the R package as a zip file.

Issue Links

• is related to: SPARK-6869 Add pyspark archives path to PYTHONPATH
• is related to: SPARK-10500 sparkr.zip cannot be created if $SPARK_HOME/R/lib is unwritable
• relates to: SPARK-11524 Support SparkR with Mesos cluster
• relates to: SPARK-8041 Consistently pass SparkR library directory to SparkR application
• links to: [Github] Pull Request #6743 (sun-rui)