[SPARK-11304] SparkR in yarn-client mode fails creating sparkr.zip

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.1
• Component/s: SparkR
• Assignee: Unassigned
• Reporter: Ram Venkatesh
• Created: 25/Oct/15 22:09
• Updated: 25/Oct/15 22:11
• Resolved: 25/Oct/15 22:11

Description

If you run sparkR in yarn-client mode and the spark installation directory is not writable by the current user, it fails with

Exception in thread "main" java.io.FileNotFoundException:
/usr/hdp/2.3.2.1-12/spark/R/lib/sparkr.zip (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at
org.apache.spark.deploy.RPackageUtils$.zipRLibraries(RPackageUtils.scala:215)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:371)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The behavior is the same with the pre-built spark-1.5.1-bin-hadoop2.6
bits also.

We need to either use an existing sparkr.zip if we find one in the R/lib directory, or create the file in a location accessible to the submitting user.

Temporary hack workaround - create a world-writable file called sparkr.zip under R/lib. It will still fail if multiple users submit jobs at the same time.