[SPARK-10500] sparkr.zip cannot be created if $SPARK_HOME/R/lib is unwritable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.6.0
• Component/s: SparkR
• Assignee: Sun Rui
• Reporter: Jonathan Kelly
• Created: 09/Sep/15 01:20
• Updated: 17/Oct/16 02:08
• Resolved: 16/Nov/15 03:29

Description

As of SPARK-6797, sparkr.zip is re-created each time spark-submit is run with an R application, which fails if Spark has been installed into a directory to which the current user doesn't have write permissions. (e.g., on EMR's emr-4.0.0 release, Spark is installed at /usr/lib/spark, which is only writable by root.)

Would it be possible to skip creating sparkr.zip if it already exists? That would enable sparkr.zip to be pre-created by the root user and then reused each time spark-submit is run, which I believe is similar to how pyspark works.

Another option would be to make the location configurable, as it's currently hardcoded to $SPARK_HOME/R/lib/sparkr.zip. Allowing it to be configured to something like the user's home directory or a random path in /tmp would get around the permissions issue.

By the way, why does spark-submit even need to re-create sparkr.zip every time a new R application is launched? This seems unnecessary and inefficient, unless you are actively developing the SparkR libraries and expect the contents of sparkr.zip to change.

Issue Links

• is related to: SPARK-8313 Support Spark Packages containing R code with --packages
• relates to: SPARK-11524 Support SparkR with Mesos cluster
• relates to: SPARK-11525 Support spark packages containing R source code in Standalone mode
• relates to: SPARK-6797 Add support for YARN cluster mode
• relates to: SPARK-9603 Re-enable complex R package test in SparkSubmitSuite
• links to: [Github] Pull Request #9390 (sun-rui)
• links to: [Github] Pull Request #10372 (sun-rui)