[SPARK-1115] Better error message when python worker process dies

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.1, 1.0.0
• Component/s: PySpark
• Assignee: Bouke van der Bijl
• Reporter: Sanford Ryza
• Created: 21/Feb/14 01:36
• Updated: 26/Feb/14 14:58
• Resolved: 26/Feb/14 14:58

Description

Right now all I see is:

java.io.EOFException
	at java.io.DataInputStream.readInt(DataInputStream.java:392)
	at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:177)
	at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:55)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:42)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:89)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:53)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
	at org.apache.spark.scheduler.Task.run(Task.scala:53)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:49)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)