Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1425

sparkr.zip is not distributed to executors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.1
    • Fix Version/s: 0.6.2
    • Component/s: None
    • Labels:
      None

      Description

      So if R daemon is required in executor, the R script will fail.

      How to reproduce it

      df <- createDataFrame(sqlContext, mtcars)
      showDF(df)
      

      Exception in executor side:

      10:16:20,024  INFO org.apache.spark.storage.memory.MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 14.2 KB, free 366.3 MB)
      10:16:21,018  INFO org.apache.spark.api.r.BufferedStreamThread:54 - Fatal error: cannot open file '/Users/jzhang/Temp/hadoop_tmp/nm-local-dir/usercache/jzhang/appcache/application_1473129941656_0037/container_1473129941656_0037_01_000002/sparkr/SparkR/worker/daemon.R': No such file or directory
      10:16:31,023 ERROR org.apache.spark.executor.Executor:91 - Exception in task 0.2 in stage 1.0 (TID 3)
      java.net.SocketTimeoutException: Accept timed out
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
          at java.net.ServerSocket.implAccept(ServerSocket.java:545)
          at java.net.ServerSocket.accept(ServerSocket.java:513)
          at org.apache.spark.api.r.RRunner$.createRWorker(RRunner.scala:367)
          at org.apache.spark.api.r.RRunner.compute(RRunner.scala:69)
          at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:49)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjffdu Jeff Zhang
                Reporter:
                zjffdu Jeff Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: