Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1413

Pyspark, can not create data frame by using sqlContext

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.6.1
    • Fix Version/s: 0.6.1
    • Component/s: Core, zeppelin-interpreter
    • Labels:
      None
    • Environment:

      docker container, ubuntu 14.04, spark 2.0, python 2.7

      Description

      I got problem when I try to use sqlContext to create a data frame in Zeppelin notebook, but works fine in pysprak console.

      l = [('alice', 1)]
      sqlContext.createDataFrame(l).collect()
      

      The Error message

      Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-8877291249558119831.py", line 266, in <module>
          raise Exception(traceback.format_exc())
      Exception: Traceback (most recent call last):
        File "/tmp/zeppelin_pyspark-8877291249558119831.py", line 264, in <module>
          exec(code)
        File "<stdin>", line 2, in <module>
        File "/opt/spark/python/pyspark/sql/context.py", line 299, in createDataFrame
          return self.sparkSession.createDataFrame(data, schema, samplingRatio)
        File "/opt/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", line 933, in __call__
          answer, self.gateway_client, self.target_id, self.name)
        File "/opt/spark/python/pyspark/sql/utils.py", line 63, in deco
          return f(*a, **kw)
        File "/opt/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py", line 316, in get_return_value
          format(target_id, ".", name, value))
      Py4JError: An error occurred while calling o49.createDataFrame. Trace:
      py4j.Py4JException: Method createDataFrame([class java.util.ArrayList, null, null]) does not exist
      	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
      	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
      	at py4j.Gateway.invoke(Gateway.java:272)
      	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
      	at py4j.commands.CallCommand.execute(CallCommand.java:79)
      	at py4j.GatewayConnection.run(GatewayConnection.java:211)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              zjffdu Jeff Zhang
              Reporter:
              robinlin robinlin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: