Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11304

SparkR in yarn-client mode fails creating sparkr.zip

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.5.1
    • Fix Version/s: None
    • Component/s: SparkR
    • Labels:
      None

      Description

      If you run sparkR in yarn-client mode and the spark installation directory is not writable by the current user, it fails with

      Exception in thread "main" java.io.FileNotFoundException:
      /usr/hdp/2.3.2.1-12/spark/R/lib/sparkr.zip (Permission denied)
      at java.io.FileOutputStream.open0(Native Method)
      at java.io.FileOutputStream.open(FileOutputStream.java:270)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      at
      org.apache.spark.deploy.RPackageUtils$.zipRLibraries(RPackageUtils.scala:215)
      at
      org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:371)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

      The behavior is the same with the pre-built spark-1.5.1-bin-hadoop2.6
      bits also.

      We need to either use an existing sparkr.zip if we find one in the R/lib directory, or create the file in a location accessible to the submitting user.

      Temporary hack workaround - create a world-writable file called sparkr.zip under R/lib. It will still fail if multiple users submit jobs at the same time.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              venkateshrin Ram Venkatesh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: