Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.7.1
-
None
-
zeppelin-0.7.1,livy-0.4-snapshot
Description
I am using Livy interpreter to interact with spark cluster. However, every time some error occurs,Zeppelin only print one line of error message which is extremely unhelpful to debug the code.
For example, If I get the following error output in Zeppelin:
<console>:113: error: overloaded method value createDataFrame with alternatives:
I can't found out exactly what caused the error.
But executing the same code on spark shell I would get the following error :
<console>:56: error: overloaded method value createDataFrame with alternatives:
(data: java.util.List[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
(rdd: org.apache.spark.api.java.JavaRDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
(rdd: org.apache.spark.rdd.RDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and>
(rows: java.util.List[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
(rowRDD: org.apache.spark.api.java.JavaRDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and>
(rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame
cannot be applied to (org.apache.spark.rdd.RDD[String], org.apache.spark.sql.types.StructType)
val testDF = spark.createDataFrame(rdds, schema)
All the detailed error log is returned by Livy server, but in another field called "traceback". In BaseLivyInterpreter.java, Zepplein only print out the "evalue" which contain only the header line of the error log.
I think print "traceback" log would be much more helpful to debug spark job when using Livy.
Attachments
Attachments
Issue Links
- links to
