Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-602

An error during 'magic' (table / json) encoding of a result

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Interpreter
    • Labels:
      None

      Description

      The error occurs during the execution of the statement:

      val x = List((1, "a"), (3, "b")).toDF("number", "key")
      %table x //or %json x

      Environment:
      Docker
      Livy: 0.6-incubating
      Spark: 2.4.0
      Hadoop: 3.0
      Dockerfile:

      FROM gettyimages/spark:2.4.0-hadoop-3.0
      
      RUN apt-get update && \
          apt-get install -y wget && \
          rm -rf /var/lib/apt/lists/*
      
      RUN mkdir -p /apps && \
          cd /apps && \
          wget http://apache.cp.if.ua/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip && \
          unzip apache-livy-0.6.0-incubating-bin.zip -d /apps && \
          rm -f apache-livy-0.6.0-incubating-bin.zip && \
          mkdir /apps/apache-livy-0.6.0-incubating-bin/logs
      
      # Expose port
      EXPOSE 8998
      
      CMD ["/apps/apache-livy-0.6.0-incubating-bin/bin/livy-server"]

      Reproducing steps (bash):

      docker build . -t livy-spark
      
      docker run -d -p 8998:8998 livy-spark
      
      curl localhost:8998/sessions -X POST --data '{"kind":"spark"}' -H "Content-Type: application/json"
      
      curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \r\n%table x"}'
      
      curl localhost:8998/sessions/0/statements/0
      
      // response
      "id":0,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \n%table x","state":"available","output":{"status":"error","execution_count":0,"ename":"Internal Error: java.lang.NoClassDefFoundError","evalue":"scala/tools/scalap/scalax/rules/scalasig/Type","traceback":[]},"progress":1.0}%
      
      curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \r\n%json x"}'
      
      curl localhost:8998/sessions/0/statements/1
      
      // response
      {"id":1,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\") \n%json x","state":"available","output":{"status":"error","execution_count":1,"ename":"ValueError","evalue":"Failed to convert value into a JSON value","traceback":[]},"progress":1.0}%

      The error:
      Livy cannot load the scala/tools/scalap/scalax/rules/scalasig/Type class.

      The line causing the error:
      https://github.com/apache/incubator-livy/blob/v0.6.0-incubating/repl/src/main/scala/org/apache/livy/repl/AbstractSparkInterpreter.scala#L186

      Stacktrace: 
      https://gist.github.com/iRevive/977a67940b05ab2faaa3f3be5b426038

      Possible root cause:
      Looks like there is a problem with the classpath.
      Currently, Livy has scalap-2.11.0.jar in the classpath. I tried to replace it with scalap-2.11.12.jar but it still doesn't work. 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              maksim.ochenashko Maksim Ochenashko
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: