Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.6.0
-
None
Description
The error occurs during the execution of the statement:
val x = List((1, "a"), (3, "b")).toDF("number", "key") %table x //or %json x
Environment:
Docker
Livy: 0.6-incubating
Spark: 2.4.0
Hadoop: 3.0
Dockerfile:
FROM gettyimages/spark:2.4.0-hadoop-3.0 RUN apt-get update && \ apt-get install -y wget && \ rm -rf /var/lib/apt/lists/* RUN mkdir -p /apps && \ cd /apps && \ wget http://apache.cp.if.ua/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip && \ unzip apache-livy-0.6.0-incubating-bin.zip -d /apps && \ rm -f apache-livy-0.6.0-incubating-bin.zip && \ mkdir /apps/apache-livy-0.6.0-incubating-bin/logs # Expose port EXPOSE 8998 CMD ["/apps/apache-livy-0.6.0-incubating-bin/bin/livy-server"]
Reproducing steps (bash):
docker build . -t livy-spark docker run -d -p 8998:8998 livy-spark curl localhost:8998/sessions -X POST --data '{"kind":"spark"}' -H "Content-Type: application/json" curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \r\n%table x"}' curl localhost:8998/sessions/0/statements/0 // response "id":0,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \n%table x","state":"available","output":{"status":"error","execution_count":0,"ename":"Internal Error: java.lang.NoClassDefFoundError","evalue":"scala/tools/scalap/scalax/rules/scalasig/Type","traceback":[]},"progress":1.0}% curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \r\n%json x"}' curl localhost:8998/sessions/0/statements/1 // response {"id":1,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \n%json x","state":"available","output":{"status":"error","execution_count":1,"ename":"ValueError","evalue":"Failed to convert value into a JSON value","traceback":[]},"progress":1.0}%
The error:
Livy cannot load the scala/tools/scalap/scalax/rules/scalasig/Type class.
The line causing the error:
https://github.com/apache/incubator-livy/blob/v0.6.0-incubating/repl/src/main/scala/org/apache/livy/repl/AbstractSparkInterpreter.scala#L186
Stacktrace:
https://gist.github.com/iRevive/977a67940b05ab2faaa3f3be5b426038
Possible root cause:
Looks like there is a problem with the classpath.
Currently, Livy has scalap-2.11.0.jar in the classpath. I tried to replace it with scalap-2.11.12.jar but it still doesn't work.