Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11304

SparkR in yarn-client mode fails creating sparkr.zip

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.5.1
    • None
    • SparkR
    • None

    Description

      If you run sparkR in yarn-client mode and the spark installation directory is not writable by the current user, it fails with

      Exception in thread "main" java.io.FileNotFoundException:
      /usr/hdp/2.3.2.1-12/spark/R/lib/sparkr.zip (Permission denied)
      at java.io.FileOutputStream.open0(Native Method)
      at java.io.FileOutputStream.open(FileOutputStream.java:270)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      at
      org.apache.spark.deploy.RPackageUtils$.zipRLibraries(RPackageUtils.scala:215)
      at
      org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:371)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

      The behavior is the same with the pre-built spark-1.5.1-bin-hadoop2.6
      bits also.

      We need to either use an existing sparkr.zip if we find one in the R/lib directory, or create the file in a location accessible to the submitting user.

      Temporary hack workaround - create a world-writable file called sparkr.zip under R/lib. It will still fail if multiple users submit jobs at the same time.

      Attachments

        Activity

          People

            Unassigned Unassigned
            venkateshrin Ram Venkatesh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: