Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11304

SparkR in yarn-client mode fails creating sparkr.zip

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.5.1
    • None
    • SparkR
    • None

    Description

      If you run sparkR in yarn-client mode and the spark installation directory is not writable by the current user, it fails with

      Exception in thread "main" java.io.FileNotFoundException:
      /usr/hdp/2.3.2.1-12/spark/R/lib/sparkr.zip (Permission denied)
      at java.io.FileOutputStream.open0(Native Method)
      at java.io.FileOutputStream.open(FileOutputStream.java:270)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      at
      org.apache.spark.deploy.RPackageUtils$.zipRLibraries(RPackageUtils.scala:215)
      at
      org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:371)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

      The behavior is the same with the pre-built spark-1.5.1-bin-hadoop2.6
      bits also.

      We need to either use an existing sparkr.zip if we find one in the R/lib directory, or create the file in a location accessible to the submitting user.

      Temporary hack workaround - create a world-writable file called sparkr.zip under R/lib. It will still fail if multiple users submit jobs at the same time.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            venkateshrin Ram Venkatesh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment