Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-205

sparkr.zip is unzipped on executor side

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.3
    • Component/s: Interpreter
    • Labels:
      None

      Description

      The following code will fail in SparkRInterpreter.

      sqlContext <- sparkRSQL.init(sc)
      df <- createDataFrame(sqlContext, faithful)
      head(df)
      

      The root cause is that sparkr.zip is not unzipped in the executor side.

      16/08/20 09:31:59 INFO r.BufferedStreamThread: Fatal error: cannot open file '/Users/jzhang/Temp/hadoop_tmp/nm-local-dir/usercache/jzhang/appcache/application_1471337400283_0082/container_1471337400283_0082_01_000002/sparkr/SparkR/worker/daemon.R': No such file or directory
      16/08/20 09:32:09 ERROR executor.Executor: Exception in task 0.0 in stage 1.0 (TID 1)
      java.net.SocketTimeoutException: Accept timed out
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
          at java.net.ServerSocket.implAccept(ServerSocket.java:545)
          at java.net.ServerSocket.accept(ServerSocket.java:513)
          at org.apache.spark.api.r.RRDD$.createRWorker(RRDD.scala:432)
          at org.apache.spark.api.r.BaseRRDD.compute(RRDD.scala:63)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      

        Attachments

          Activity

            People

            • Assignee:
              zjffdu Jeff Zhang
              Reporter:
              zjffdu Jeff Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: